KR102348869B1 - Method and apparatus for prediting medical image using conditional generative adversarial network - Google Patents

Abstract

The present invention relates to a method and apparatus for prediction of a medical image using a conditionally generated adversarial network. A method according to an embodiment of the present invention uses a conditional generative adversarial network (cGAN) and is a method for predicting a medical image performed in a computing device or a computing network, and a plurality of adjacent two-dimensional medical images and a generator that receives a condition and generates and outputs other 2D medical images according to the condition, receives a 2D medical image and a condition, and determines whether the condition matches the 2D medical image and slices the actual 2D medical image A first discriminator that determines whether or not the 2D medical images are adjacent in one axial direction and a second discriminator that determines whether the actual 3D image slice matches the 2D medical images by adversarial learning learning stage; and a prediction step of generating a 3D medical image predicted for the condition by inputting a plurality of adjacent 2D medical images and conditions that are objects to the learned generator.

Description

Method and apparatus for prediction of medical images using conditionally generated adversarial networks

The present invention relates to a method and apparatus for predicting a medical image, and more particularly, to a method and apparatus for predicting a medical image using a conditionally generated adversarial network.

Alzheimer's disease (AD) is a degenerative neurological disease in which abnormal proteins such as amyloid beta protein and tau protein accumulate in the brain and brain neurons slowly die. This Alzheimer's disease progresses in the order of Normal Condition (NC), Mild Cognitive Impairment (MCI), and AD, and the degree of progression is measured through medical images such as Magnetic Resonance Image (MRI). can be identified more accurately. Hereinafter, a stage related to the progression of a certain disease, such as NC, MCI, and AD, is referred to as a “disease progression stage”.

Recently, various techniques for processing and predicting medical images by applying various machine learning algorithms have been developed. However, in order to accurately predict a medical image such as a Magnetic Resonance Imaging (MRI) image, a large amount of medical image data is required when a predictive model is trained, but the amount of medical image data is limited due to its characteristics. has no choice but to work

In order to solve the problem of this limitation of medical image data, a study on synthetic image data generation based on a generative adversarial network (GAN) has been proposed. For example, a GAN model may be applied to generate a Computed Tomography (CT) image from an MRI image, a Positron Emission Tomography (PET) image from an MRI image, and a PET image from a CT image, respectively. . However, this method is a method in which domain conversion is performed only between two predefined domains. Therefore, it is impossible for the GAN to generate only an image corresponding to a specific condition.

Meanwhile, in order to solve the limitations of GAN, a conditional generative adversarial network (cGAN) that can transform a source image into a domain of a target condition by applying a specific target condition has been newly developed. can be suggested. That is, by using cGAN, a source image can be converted into an image corresponding to a specific condition.

However, in the case of the prior art using cGAN, it is only a technology for generating a corresponding medical image according to an age condition, and a technology for generating a corresponding medical image according to a condition of a specific disease stage such as NC, MCI, or AD is not available. no. In addition, the prior art is a technology related to two-dimensional (2D) image generation, which has a large amount of data to be processed, so it is difficult to optimize a model that directly generates a three-dimensional (3D) image. In particular, when a 3D image is reconstructed by connecting 2D images generated through the prior art, since each 2D image is independently generated, it is discontinuous across other axes that are not used during training of the 2D GAN model. The part appears to cause problems.

In order to solve the problems of the prior art as described above, the present invention provides a method for predicting and generating a medical image of another disease progression stage based on a medical image of a certain disease stage using a conditionally generated adversarial network (cGAN); The purpose is to provide a device.

Another object of the present invention is to provide a method and apparatus for predicting and generating a medical image, which can reconstruct a continuous 3D medical image across each axis of 3D with a small memory requirement.

However, the problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. There will be.

A method according to an embodiment of the present invention for solving the above problems uses a conditional generative adversarial network (cGAN), and is a method for predicting a medical image performed in a computing device or a computing network. , (1) a generator that receives a plurality of adjacent two-dimensional medical images and conditions and generates and outputs other two-dimensional medical images according to the corresponding conditions; A first discriminator that determines whether conditions match and whether a slice of a real 2D medical image is matched; A learning step of adversarially learning a second discriminator to determine whether or not, (2) a prediction step of generating a 3D medical image predicted for the condition by inputting a plurality of adjacent 2D medical images and conditions that are objects to the learned generator includes

The condition may include any one of disease progression stages related to disease progression.

The medical image may include a brain measurement image, and the disease may include Alzheimer's disease.

The disease progression stage may include a Normal Condition (NC) stage, a Mild Cognitive Impairment (MCI) stage, and an Alzheimer's Disease (AD) stage.

In the learning step, after the first and second discriminators are learned, the process of learning the generator may be repeatedly performed. The apparatus according to an embodiment of the present invention implements a conditional generative adversarial network (cGAN). A device for predicting a medical image performed in a computing device or a computing network, wherein the cGAN (1) receives a plurality of adjacent two-dimensional medical images and conditions and generates other two-dimensional medical images according to the corresponding conditions (2) a first discriminator that receives a 2D medical image and condition and determines whether the corresponding condition for the 2D medical image and the actual 2D medical image slice match, (3) and a second discriminator configured to receive adjacent 2D medical images in one axial direction and determine whether the 2D medical images match an actual 3D image slice.

In the cGAN, the first discriminator and the second discriminator may be adversarially learned with respect to the generator.

The learned generator may generate a plurality of adjacent two-dimensional medical images that are objects and a three-dimensional medical image predicted for a corresponding condition when a condition is input.

The present invention configured as described above has an advantage in that it is possible to accurately predict and generate a medical image of a different disease progression stage based on a medical image of a certain disease stage using cGAN.

In addition, the present invention has the advantage of predicting and generating a medical image that can reconstruct a continuous 3D medical image across each 3D axis with a small memory requirement.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

1 shows an apparatus 100 for predicting a medical image according to an embodiment of the present invention.
2 is a block diagram of a conditional generative adversarial network 200 according to an embodiment of the present invention.
3 illustrates an operating principle of a conditionally generated adversarial network 200 according to an embodiment of the present invention.
4 is a flowchart of a prediction method according to an embodiment of the present invention.
6 shows a composite image generated using the cGAN 200 of the present invention for the hippocampus, ventricle, and gray matter region.
7 shows a synthesized image generated using the cGAN 200, 2D cGAN, and 3D cGAN of the present invention, and an original image thereof, respectively.

The above object and means of the present invention and its effects will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily understand the technical idea of the present invention. will be able to carry out In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form as the case may be, unless otherwise specified in the phrase. In this specification, terms such as “include”, “provide”, “provide” or “have” do not exclude the presence or addition of one or more other components other than the mentioned components.

In this specification, terms such as “or” and “at least one” may indicate one of the words listed together, or a combination of two or more. For example, “or B” and “at least one of B” may include only one of A or B, or both A and B.

In the present specification, descriptions according to “for example” and the like may not exactly match the information presented, such as recited properties, variables, or values, tolerances, measurement errors, limits of measurement accuracy, and other commonly known factors The embodiments of the present invention according to various embodiments of the present invention should not be limited by effects such as modifications including .

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in between. It should be understood that there is On the other hand, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In this specification, when it is described that a certain element is 'on' or 'in contact with' another element, it may be directly in contact with or connected to the other element, but another element may exist in the middle. It should be understood that On the other hand, when it is described that a certain element is 'directly on' or 'directly' of another element, it may be understood that another element does not exist in the middle. Other expressions describing the relationship between the elements, for example, 'between' and 'directly between', etc. may be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various components, but the components should not be limited by the above terms. In addition, the above terms should not be construed as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first component' may be referred to as a 'second component', and similarly, a 'second component' may also be referred to as a 'first component'.

Unless otherwise defined, all terms used herein may be used with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows an apparatus 100 for predicting a medical image according to an embodiment of the present invention. Also, FIG. 2 shows a block diagram of a conditional generative adversarial network 200 according to an embodiment of the present invention.

The prediction apparatus 100 according to an embodiment of the present invention predicts a medical image of another disease progression stage based on a medical image of a specific disease stage using a conditional generative adversarial network (cGAN) 200 . It may be a device for generating (synthesizing) The prediction apparatus 100 may perform a prediction operation on data of a medical image obtained by the image acquisition apparatus.

For example, the image acquisition device includes an X-ray device, an ultrasound device, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, and a positron emission tomography (PET) device. ) device, and the like, and the medical image may be an X-ray image, an ultrasound image, a CT image, an MRI image (or MR image), a PET image, or the like, but is not limited thereto.

The prediction device 100 may be a computing device included in the image acquisition device, and may be a separately provided computing device or computing network. Referring to FIG. 1 , the prediction apparatus 100 may include an input unit 110 , a communication unit 120 , a display 130 , a memory 140 , a processor 150 , and the like.

The input unit 110 generates input data in response to a user's input regarding a condition or the like. The input unit 110 may include various input means. For example, the input unit 110 may include a keyboard, a keypad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), a menu button (menu button) and the like may be included, but is not limited thereto.

The communication unit 120 is a component that communicates with an external device such as an image acquisition device and a server, and may transmit/receive medical image data and the like. For example, the communication unit 120 is 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energe (BLE), near field communication (NFC), Wireless communication such as Wi-Fi communication may be performed, and wired communication such as cable communication may be performed, but is not limited thereto.

The display 130 displays various image data such as medical images on the screen. Also, the display unit 130 may be combined with the input unit 120 to be implemented as a touch screen or the like. For example, the display unit 130 may be configured as a non-emissive panel or a light-emitting panel, but is not limited thereto.

The memory 140 stores various types of information necessary for the operation of the prediction apparatus 100 . For example, the storage information may include, but is not limited to, medical image data, information related to cGAN, program information related to a prediction method to be described later, and the like.

The processor 150 may perform various control operations of the prediction apparatus 100 . For example, the processor 150 may control execution of a prediction method to be described later, and the remaining components of the prediction apparatus 100 , that is, the input unit 110 , the communication unit 120 , the display 130 , and the memory 140 . , the operation of the processor 150 and the like may be controlled, but the present invention is not limited thereto.

Meanwhile, the cGAN 200 according to an embodiment of the present invention used for prediction of a medical image, as shown in FIG. 2 , includes a generator G, a first discriminator, or a 2D discriminator. F), and a second discriminator (or 3D discriminator) H. These can be subjected to adversarial learning, and the learned models are classified as 'generic model', 'first discriminator', ' It may also be referred to as a 'second discriminant model'.

Information related to the cGAN 200 may be stored in the memory 140 , and may be performed by the processor 150 through at least one process. However, the generator (G), the first discriminator (F), and the second discriminator (H) will be described in more detail in the prediction method to be described later.

Hereinafter, a prediction method according to an embodiment of the present invention controlled by the processor 150 will be described.

However, in the following, any disease progression of Alzheimer's Disease (AD) using the cGAN 200 is used in Normal Condition (NC), Mild Cognitive Impairment (MCI) and AD. A method of predicting and generating a medical image of another disease progression stage based on a medical image of the stage (MRI image of the brain) will be described, but the present invention is not limited thereto. That is, the present invention can also be applied to a method of predicting and generating a medical image of another disease progression stage based on a medical image of a disease stage other than Alzheimer's disease. In this case, the medical image may be an image obtained by photographing a part other than the image obtained by photographing the brain.

Also, hereinafter, 'image' may refer to a medical image, and 'synthetic image' may refer to an image generated by the generator (G). In addition, the slice of the actual 2D image is expressed as 'x2D', the slice of the synthesized 2D image is expressed as 's2D', the slice of the actual 3D image is expressed as 'x3D', and the slice of the synthesized 3D image is expressed as 's3D', respectively. can be

3 illustrates an operating principle of a conditionally generated adversarial network 200 according to an embodiment of the present invention.

The generator G is a component for synthesizing 2D images. That is, the generator G generates an s2D corresponding to a target condition c (ie, NC, MCI, or AD) from an input 2D image (x2D or s2D). In particular, the generator G generates an adjacent s2D from a plurality of adjacent 2D images and condition (c), thereby enabling synthesis of a 3D image related thereto. This generator (G) is trained by adversarial learning to modify the brain regions affected by the disease, while reversing the transformation through cycle learning. maintain identity.

The generator G may include an encoder, a transition layer, and a decoder. The encoder reduces the feature map to 1/4 of the original size of x2D through three convolutional layers. Then, the transition layer processes the feature map through 6 residual blocks. At this time, the condition (c) and the feature map are combined. In the final stage of the generator G, the decoder defined by three transpose convolution layers transforms the latent representation into s2D of the same size as x2D.

The first discriminator F is configured to ensure that realistic 2D image slices satisfying the target condition are synthesized. That is, the first discriminator F determines whether the label (c) of the condition for the input 2D image matches, and whether the condition matches the x2D (ie, whether the input 2D image corresponds to x2D or s2D). to determine whether it corresponds to ).

The first discriminator F may be composed of a convolutional neural network (CNN) including five 2D convolutional layers and two fully connected layers. Two fully connected layers perform an actual classification (Fa) for a 2D image (ie, to classify whether it is real or synthetic) and a classification (Fb) for a condition (ie, to classify whether it is NC, MCI, or AD). . The input of F is axial x2D or s2D.

The second discriminator H is configured to ensure a 3D structure while solving a problem of a memory limitation that occurs when generating and discriminating a 3D image. That is, the second discriminator H may help to generate a smoothing 3D image.

In the generator G, in consideration of the smoothness of images synthesized in 3D space, k (however, k is a natural number greater than or equal to 2) is sequentially arranged in the axial direction. That is, the adjacent images which are _2D ¹ s, ² s _2D, ..., s ^k for each _2D image on the mini-batch (mini-batch) is generated. The second discriminator H determines whether the adjacent images correspond to x3D (ie, whether they correspond to x3D or s3D). Accordingly, the second discriminator H may improve the image quality of the 3D space, thereby solving the problem of discontinuous portions appearing across a plurality of axes in the prior art. That is, the present invention makes it possible to reconstruct a continuous 3D image across each axis of 3D through the second discriminator (H). In this case, x3D is a concatenation of adjacent real 2D images x ¹ _2D , x ² _2D , ..., x ^k _2D , and s3D is adjacent synthetic 2D images s ¹ _2D , s ² _2D , .. ., s ^k _2D is concatenated.

Like the first discriminator (F), the second discriminator (H) may be configured as a CNN. However, the CNN of the second discriminator H consists of five 3D convolutional layers and one fully connected layer whose input distinguishes whether adjacent 2D slices correspond to x3D or s3D. Accordingly, the second discriminator H can ensure that the model generates a 3D image of excellent quality in all three axes of 3D. That is, referring to FIG. 2 , it can be seen that s3D is reconstructed by concatenating s2D, which is k consecutive 2D slices.

In summary, the second discriminator (H) performs an operation on adjacent 2D slices in the mini-batch concatenated and generated by the generator (G). According to the second discriminator H, the generator G is trained to synthesize 2D images in consideration of information in 3D space.

4 is a flowchart of a prediction method according to an embodiment of the present invention.

Referring to FIG. 4 , the prediction method according to an embodiment of the present invention includes S10 and S20.

First, in S10, the generator (G), the first discriminator (F), and the second discriminator (H) are subjected to adversarial learning. At this time, in 2D and 3D space, the cGAN 200 is trained using an _{adversarial loss function (L 2D-adv} , L _{2D-adv ) as shown in (1) and (2) below.} can

That is, L _2D-adv is a hostile loss function for a 2D image, and L _3D-adv is a hostile loss function for a 3D image. _{The first term related to L 2D-adv} in Equation (1) is a function that outputs a probability value close to 1 when a real 2D image (x2D) is input to the first discriminator (F), and the second term is the second term 1 This is a function that outputs a probability value close to 0 when a fake 2D image s2D is input to the discriminator F. _{The first term related to L 3D-adv} in Equation (2) is a function that outputs a probability value close to 1 when an actual 3D image (ie, continuous 2D images) (x3D) is input to the second discriminator (H). , the second term is a function that outputs a probability value close to 0 when a fake 3D image (ie, consecutive 2D images) (s3D) is input to the second discriminator H.

s2D must be similar to x2D and must satisfy target condition (c). To solve this, Fb classifies conditions for the input 2D image slice. Meanwhile, the condition predicted for s2D is used to optimize the second discriminator (G), and the condition predicted for x2D is used to optimize the first discriminator (F). Accordingly, the classification loss for the real image (L ^r _cls ) and the loss for the synthetic image (L ^s _cls ) are defined as follows (3) and (4).

That is, in Equation (3), L ^r _cls is a value obtained by taking the -log value of the probability of matching the condition (c) when an actual 2D image slice (x2D) is given to the first discriminator (F). If the corresponding condition is met, a value of 0 is obtained, and if it is incorrect, a value greater than that value is obtained. In equation (4), L ^s _cls is When a fake 2D image slice (s2D) is given to the first discriminator (F), it can be expressed as a value obtained by taking the -log value of the probability of matching the condition. value greater than that.

Since the synthesized image must maintain the identity of the input image, a cycle consistency loss (L _cyc ) is added along with the adversarial loss. As shown in the following (5), the cyclic coherence loss (L _cyc ) returns the transformation using the original condition (c') of x2D and the generator (G) using s2D.

That is, in Equation (5), L _cyc is the real 2D image (x2D), the fake 2D image (s2D) and the condition (c'), which is an image generated by inputting the generator (G), G(s2D, c') It can be expressed as a difference between

Finally, as in the following (6) to (8), the generator (G), the first discriminator (F) and the second discriminator (H) have respective loss functions (L _G , L _F and L _H ) is learned while optimizing.

At this time, simultaneous learning means that learning is carried out in consideration of all losses, meaning that learning is carried out by alternating the first and second discriminators (F, H) and the generator (G). That is, after the first and second discriminators (F, H) are first learned, the generator (G) is learned based on the learned first and second discriminators (F, H), and again the first and second discriminators (F, H) are trained. 2 The discriminators F and H are updated and the generator G is updated, and this process may be repeated. For example, this process is performed in a direction in which each loss function (L _G , L _F and L _H ) is set to a predetermined appropriate value, or in a direction to decrease _{each loss function (L G} , L _F and L _{H ),} Alternatively, learning may be progressed and updated in a direction that minimizes each of the loss functions L _G , L _F and L _{H , but is not limited thereto.}

In addition, parameters for optimizing each loss effect in (6) to (8) can be applied. That is, cls and λ _cyc are hyperparameters for balancing loss effects, respectively, and cls may be 1 and λ _cyc may be 0.1, but is not limited thereto.

Thereafter, in S20, it is a step of applying the optimally learned cGAN 200 to an actual target image. That is, in S20 , by inputting the target 2D image and the condition to the optimally learned generator G, the 2D image predicted for the condition may be generated.

For example, if an MRI image of a patient to be tested for Alzheimer's disease and its condition (NC, MCI, or AD) are input to the pre-learned generator (G), the generator (G) will generate the corresponding condition (NC, MCI, or AD). A predicted MRI image may be generated. In particular, if a plurality of 2D images and conditions adjacent to the learned generator G are input, adjacent 2D images predicted for the corresponding condition can be generated, and 3 related 3D images are generated through the resultant adjacent 2D images. A dimensional image may be synthesized.

<Experiment result>

1) Dataset

The dataset from the Alzheimer's Disease Neuroimaging Initiative (ADNI) was used. 200 MRI images of AD subjects, 400 MRI images of MCI subjects, and 200 MRI images of NC subjects, a normal control group, were collected. Images were aligned and resized using affine transformation so that the size of all images was 192 × 192 × 160.

2) Implementation details

The data were divided into two even subsets (set A and set B). Using the model of the present invention, a composite image (set A') with an axial view of an MR image of size 192 × 192 in set A was generated. For each image, images were generated for each of the other two conditions except for the original condition (c') (for example, NC images and MCI images are generated from AD images, and MCI images and AD images are generated from NC images) ). In order to train the generator (G), the first discriminator (F), and the second discriminator (H) at the same time, k consecutive 2D slices from the MRI image are sampled and a training mini A training mini-batch was constructed. In this experiment, each case where k was a value of 3, 6, and 8 was evaluated. The model was trained using the Adam Optimizer with 150,000 iterations and a learning rate of 0.0001 with constant attenuation. The program code was developed based on the PyTorch deep learning framework and was run on a GTX TITAN X NVIDIA GPU.

3) Evaluation setting

In order to quantitatively measure the quality of the generated 3D image, FID (Frechet Inception Distance) was measured between set A' and set B. FID evaluates the difference between the real image and the embedded representation of the generated image, and measures their mean and covariance differences. The embedded representation was computed by encoding the image as a second layer of the pretrained Inception network on the ImageNet data set. To apply the pre-trained 2D Inception network to 3D images, 2D image slices were extracted from 3 axes, and then the FID score was measured for each axis.

In addition, GAN-train and GAN-test scores were also measured. The FID score assumes that the real and generated images have a Gaussian distribution similar to the images in the ImageNet dataset, but this may not be a good representation of biomedical imaging. However, the distribution similarity between actual and generated biomedical images can still be measured by GAN-train and GAN-test scores. To calculate the GAN-train score, a classification model was trained on the generated data and then tested on the real data. In the case of GAN-test, the model was trained with real data and then the generated data was tested. If each discriminant model is trained and tested on real data and the GAN-train and GAN-test scores have almost similar performance, the corresponding GAN model is considered effective. We measured the scores for classification for three conditions (NC vs MCI vs AD) and binary classification between NC and AD (ie, NC vs AD), which is a relatively easy task.

5 shows a network architecture for classification.

For classification models, see “Pan, Y., Liu, M., Lian, C., Zhou, T., Xia, Y., Shen, D.: Synthesizing Missing PET from MRI with Cycle-consistent Generative Adversarial Networks for Alzheimer's Disease Diagnosis. In: Medical Image Computing and Computer Assisted Intervention. pp. 455{463 (2018)”. In particular, 27 patches with a size of 48 × 48 × 48 were extracted from various locations in the entire 3D volume. After encoding the patch into an embedded representation, all embedded vectors were concatenated (concatenated), and a single classification result for the 3D image was finally measured. Classification model details may be as shown in FIG. 5 . However, in addition to this, any type of classification model may be used, and the accuracy may vary depending on the corresponding model used.

The cGAN 200 of the present invention was compared with 2D cGAN and 3D cGAN. In this case, the 2D cGAN has a structure similar to that of the cGAN 200 of the present invention, but the second discriminator G is not used. The 3D cGAN also has a structure similar to that of the cGAN 200 of the present invention, but uses a 3D generator that generates a 3D image. However, due to memory limitations, the 3D generator of 3D cGAN was implemented to generate 20 adjacent 2D images at a time instead of the entire 3D image. For fair comparison, the same experiment was performed using 2D cGAN and 3D cGAN having the same hyperparameters used in the present invention. In addition, FID and GAN-train scores were measured between set A and set B to indicate the upper limit.

4) Quantitative results

NC vs MCI vs AD NC vs AD Models FID(A) FID(C) FID(S) GAN-Train GAN-Test GAN-Train GAN-Test real video 0.86 1.04 0.84 49.00 - 74.68 - 2D cGANs 34.30 40.36 46.36 43.67 42.27 56.32 75.95 3D cGAN 60.74 144.26 134.72 37.16 29.43 50.00 47.26 Our (k=3) 42.66 49.44 52.33 43.00 45.88 71.52 82.09 Our (k=6) 44.79 37.18 30.52 45.33 52.74 72.79 79.76 Our (k=8) 30.70 26.82 27.25 50.00 52.74 73.42 80.32

Table 1 shows the image quality scores of images generated by the cGAN 200 of the present invention and 2D and 3D cGANs as comparison objects. At this time, FID scores were calculated in the axial (A), coronal (C) and sagittal (S) views, respectively. In addition, GAN-Train and GAN-Test scores were calculated for classification for three conditions (NC vs MCI vs AD) and binary classification for NC and AD (NC vs AD). 2D cGAN is axial A good FID score was achieved in the orientation, but the FID score was relatively high in the coronal and sagittal views. In addition, 2D cGAN had low or inconsistent GAN-train and GAN-test scores. 3D cGAN achieved the worst results. This is because the 3D model is trained with a limited mini-batch size using a small amount of data, so the image-generating model is not properly trained. On the other hand, in the model with k = 8 of the present invention, the 3-axis FID score and the GAN-train/GAN-test score in classification for three conditions (NC vs MCI vs AD) and binary classification (NC vs AD) were 2D and 3D cGAN. This indicator indicates that the present invention is a method of modeling a basic distribution of real data with a score similar to real data (real image), which is significantly improved over 2D and 3D cGAN, which may correspond to the prior art.

5) Qualitative results

6 shows a composite image generated using the cGAN 200 of the present invention for the hippocampus, ventricle, and gray matter region. That is, in FIG. 6 , the upper part is NC, the middle part is MCI, and the lower part is AD condition, and the difference between important parts of each image is indicated by a yellow arrow.

7 shows a synthesized image generated using the cGAN 200, 2D cGAN, and 3D cGAN of the present invention, and an original image thereof, respectively. That is, in FIG. 7 , the uppermost row shows a composite image using the cGAN 200 of the present invention, the second row shows a 2D cGAN, and the third row shows a synthesized image using the 3D cGAN, and the bottom row shows the original image.

Qualitative results are shown in FIGS. 6 and 7 . That is, it was confirmed that the cGAN (200) model according to the present invention accurately predicts that the hippocampus and gray matter become noticeably smaller in AD patients than in NC patients. In particular, the synthesized image of the present invention has fewer artifacts than the synthesized image of 2D and 3D cGAN models. The results from the axial perspective are mostly satisfactory in all methods, but there is a big difference between the results from the coronal or sagittal perspective. That is, in terms of the corresponding axis, the model of the cGAN 200 according to the present invention generates a better 3D image showing sharper boundaries than the 2D and 3D cGAN models.

6) Effect on data augmentation

To understand the effect on the image generated for data magnification, separate AD and NC (AD vs NC), separate MCI and AD (MCI vs AD), and separate NC and MCI (NC vs MCI) Each binary classification model was implemented. At this time, 6 classification models were trained, 3 trained only with real data (set A), 3 were trained with augmented data and real data (set A + set A'), and then tested on set B. did. The results are shown in Table 2 below.

AD vs NC MCI vs AD NC vs MCI real data 78.48 62.90 64.25 real data +
synthesized data 81.25 63.35 66.06

That is, Table 2 shows the binary classification performance of each of the classification models obtained from the real image data (real data) and the real and synthetic image data (synthetic data). The augmented data achieved a consistent improvement of about 2% across all combinations of binary classification settings. Since the difference between NC and AD images is relatively clear, higher classification accuracy was derived. On the other hand, the classification of MCI from NC or AD has relatively low accuracy due to ambiguity.

As described above, the present invention has the advantage of predicting and generating a medical image of another disease progression stage based on a medical image of one disease stage using the cGAN 200 . That is, an MRI image of the disease stage (NC, MCI, or AD) of the target condition of AD may be generated with respect to the input MRI image. In addition, the present invention has the advantage of predicting and generating a medical image that can reconstruct a continuous 3D medical image across each 3D axis with a small memory requirement.

In particular, the main contributions of the present invention are summarized as follows.

(1) A cGAN 200 capable of predicting a change (eg, brain change, etc.) of a measurement site from a current disease stage state to another disease stage state is presented. That is, the present invention proposes a conditionally generated adversarial network (cGAN) capable of synthesizing NC, MCI, and AD images of Alzheimer's disease from any one MR image and a target condition. Thus, the present invention can play an important role in training accurate predictive models with small data sets. This model according to the present invention can help healthcare professionals better understand how a patient's condition progresses and focus their efforts on areas of the brain that can potentially suffer significant damage as the disease progresses.

(2) By adding the second discriminator (H) to the 2D cGAN structure having the generator (G) and the first discriminator (F), the generator (G) considers the structural information of the 3D space to generate a smooth and natural 3D image. Matching images can be synthesized. That is, the model according to the present invention can efficiently generate consistent high-quality 3D images in all directions in 3D space.

(3) Efficient 3D image creation is possible by using less memory for 2D images. In addition, the present invention is a consistent data augmentation method that improves classification accuracy to identify AD in MR brain images. (4) In the present invention, the effect on accuracy, etc. is demonstrated through experimental results using various measurements commonly used to evaluate the performance of GANs for 2D image generation without ground truth. In particular, by comparing the GAN-train and GAN-test scores, it is proved that the synthesized image generated by the model according to the present invention is similar to the characteristics of the real image.

In the detailed description of the present invention, although specific embodiments have been described, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and their equivalents.

100: predictive device 200: conditionally generated adversarial network (cGAN)
G: generator F: first discriminator
H: second discriminator

Claims (8)

A method for prediction of a medical image performed in a computing device or computing network using a conditional generative adversarial network (cGAN), the method comprising:
The conditionally generated adversarial network is
a generator for receiving a plurality of adjacent two-dimensional input medical images and a target condition indicating a specific stage among disease progression stages, and generating and outputting two-dimensional output medical images according to the target condition; a first discriminator configured to receive the target condition and determine whether the 2D output medical images match the target condition and whether the 2D output medical images match an actual 2D medical image slice; a second discriminator for receiving two-dimensional medical images adjacent in one axial direction among medical images and determining whether adjacent two-dimensional medical images in one axial direction correspond to an actual three-dimensional image slice;
The method is
Whether the 2D output medical images match the target condition, whether the 2D output medical images correspond to an actual 2D medical image slice, and whether the 2D medical images adjacent in the one axial direction are the actual 3D image slices a learning step of adversarially learning the generator, the first discriminator, and the second discriminator based on a loss function related to whether ? and
a prediction step of generating a 3D medical image predicted for the target condition by inputting a plurality of 2D medical images adjacent to the learned generator and the target condition;
How to include.
delete According to claim 1,
The plurality of 2D input medical images and the 2D output medical images include brain measurement images, and the disease includes Alzheimer's disease.
4. The method of claim 3,
The disease progression stages include a Normal Condition (NC) stage, a Mild Cognitive Impairment (MCI) stage, and an Alzheimer's Disease (AD) stage.
According to claim 1,
In the learning step, after the first discriminator and the second discriminator are learned, the process of learning the generator is repeatedly performed.
A device for prediction of a medical image using a conditional generative adversarial network (cGAN) and performed in a computing device or computing network, comprising:
a generator for receiving a plurality of adjacent two-dimensional input medical images and a target condition indicating a specific stage of disease progression, and generating and outputting two-dimensional output medical images according to the target condition; a first discriminator configured to receive the target condition and determine whether the 2D output medical images match the target condition and whether the 2D output medical images match an actual 2D medical image slice; a second discriminator for receiving two-dimensional medical images adjacent in one axial direction among medical images and determining whether adjacent two-dimensional medical images in one axial direction correspond to an actual three-dimensional image slice;
The apparatus determines whether the 2D output medical images match the target condition, whether the 2D output medical images match an actual 2D medical image slice, and determines whether the 2D medical images adjacent in the one axis direction are actually 3D An apparatus for adversarially learning the generator, the first discriminator, and the second discriminator based on a loss function related to whether or not a dimensional image slice is matched.
7. The method of claim 6,
The cGAN is an apparatus in which the first discriminator and the second discriminator are adversarially learned with respect to the generator.
8. The method of claim 7,
The learned generator generates a plurality of adjacent 2D medical images and a 3D medical image predicted for the target condition when the target condition is input.

Images