KR102359474B1 - Method for missing image data imputation using neural network and apparatus therefor - Google Patents

Abstract

Disclosed are a method and apparatus for replacing missing image data using a neural network. A method of replacing missing image data according to an embodiment of the present invention includes: receiving input image data for at least two or more domains among multiple preset domains; and reconstructing missing image data of a preset target domain using a neural network receiving the two or more input image data as inputs, wherein the neural network converts real image data of at least two or more of the multiple domains. The fake image data of the first target domain generated as an input and the real image data are combined, and the image restored by inputting the combined image data and the real image data must be similar to each other by using a multi-cycle coherence loss. can be trained

Description

Method and device for replacing missing image data using a neural network

The present invention relates to a technique for replacing missing image data using a neural network, and more particularly, to missing image data capable of reconstructing missing image data of a target domain using a neural network using image data of each of multiple domains as an input. It relates to an alternative method and an apparatus therefor.

Many image processing and computer vision applications require multiple input image sets to produce the desired output. For example, in brain magnetic resonance imaging (MRI), MR images having T1, T2, and FLuid-attenuated inversion recovery (FLAIR) contrast are all required for accurate diagnosis and segmentation of cancer margins. When generating a 3D volume from a multi-view camera image, most algorithms require a set of pre-determined view angles. However, it is often difficult to obtain a complete set of input data due to acquisition cost and time, and systematic errors of the data set. For example, when generating a synthetic MR contrast using a Magnetic Resonance Image Compilation sequence, a systematic error exists in the synthetic T2-FLAIR contrast image, which often leads to misdiagnosis. Missing data can also introduce significant biases, creating errors in data processing and analysis and reducing statistical efficiency.

Rather than reacquiring all data sets in unexpected situations that are often not feasible in clinical settings, missing data are sometimes replaced with substituted values, and this process is called imputation. . When all missing values are replaced, the data set can be used as input to standard techniques designed for complete data sets.

There are several standard methods for imputing missing data based on modeling assumptions for the entire set, such as mean imputation, regression imputation, and statistical imputation. However, these standard algorithms have limitations for high-dimensional data such as images, because image replacement requires knowledge of high-dimensional data manifolds.

The image-to-image conversion problem has a similar technical problem, the goal of which is to replace certain aspects of a given image with another. Tasks such as super resolution, denoising, deblurring, style transfer, semantic segmentation, and depth prediction can be performed in one domain. It may be image mapping to a corresponding image in another domain. Here, each domain has different aspects such as resolution, facial expression, and light angle, and it is necessary to know about the intrinsic manifold structure of the image data set to be converted between domains. Recently, this task has been greatly enhanced by Generative Adversarial Networks (GANs).

A general GAN framework consists of two neural networks, a generator G and a discriminator D. If the discriminator finds a feature to distinguish the fake from the real sample through training, the generator learns how to remove and synthesize the features that the discriminator uses to judge the fake from the real sample. Thus, GANs can generate more realistic samples in which the discriminator cannot distinguish between real and fake. GANs are showing remarkable performance in various computer vision tasks such as image generation and image transformation.

Unlike conventional GANs, conditional GANs (Co-GANs) control the output by adding some information labels as parameters of additional generators. Here, the generator learns to produce a fake sample with a specific condition or characteristic (a label or more detailed tag associated with the image) instead of generating a generic sample from an unknown noise distribution. A successful application of conditional GAN is for image-to-image conversions such as pix2pix for paired data and CycleGAN for unpaired data.

CycleGAN (J.-Y. Zhu, T. Park, P. Isola, and AA Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. arXiv preprint, 2017.) and DiscoGAN (T. Kim, M. Cha, H. Kim, JK Lee, and J. Kim. Learning to discover cross-domain relations with generative adversarial networks. arXiv preprint arXiv:1703.05192, 2017.) preserves key properties between input and output images by exploiting cyclic coherence loss. try to do However, such a framework can only learn the relationship between two different domains at once. This approach can handle multiple domains because each domain pair requires a separate domain pair and a total of N × (N−1) generators to process N distinct domains, as shown in Fig. 1a. When there is a scalability limit. In order to generalize the idea of multi-domain translation, the prior art has proposed a so-called StarGAN, which can learn translation mapping between multiple domains with a single generator, as shown in FIG. 1B, and a similar multi-domain transport network has recently been proposed. There has also been

StarGAN (Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. StarGAN: Unified generative adversarial networks for multidomain image-to-image translation. arXiv preprint, 1711, 2017.) and Radial GAN (J. Yoon, J. Jordon, and M. van der Schaar. RadialGAN: Leveraging multiple datasets to improve target-specific predictive models using generative adversarial networks. arXiv preprint arXiv:1802.06403, 2018.) It is a recent framework that handles multiple domains using constructors. For example, in StarGAN, a deep connection from the input image and the mask vector representing the target domain helps to map the input to the image reconstructed in the target domain. Here, the discriminator should be designed to play another role for domain classification. Specifically, the discriminator determines not only the authenticity of the sample, but also the class of the sample.

This GAN-based image transmission technology is closely related to image data replacement because image transformation can be regarded as a process capable of estimating a missing image database by modeling an image manifold structure. However, there is a fundamental difference between image imputation and translation. For example, CycleGAN and StarGAN are interested in transmitting one image to another without considering the remaining domain data sets as shown in FIGS. 1A and 1B . However, in the image replacement problem, missing data is not frequently generated, and the goal is to estimate the missing data by using another clean data set.

Embodiments of the present invention provide a method and apparatus for replacing missing image data that can improve restoration performance by reconstructing missing image data of a target domain using a neural network using image data of each of multiple domains as an input. to provide.

A method of replacing missing image data according to an embodiment of the present invention includes: receiving input image data for at least two or more domains among multiple preset domains; and reconstructing missing image data of a preset target domain by using a neural network to which the two or more input image data are input.

The neural network combines the fake image data of a first target domain generated by inputting at least two or more real image data from among the multiple domains as an input, and the real image data, and receives the combined image data as an input to restore the image data. The image and the real image data can be trained using a multi-cycle coherence loss that must be similar.

The receiving may include receiving input image data for the at least two domains and information on the target domain together.

The neural network includes a generative adversarial network (GAN), a convolutional neural network, a neural network based on a convolution framelet, and a pooling layer and an unpooling layer. It may include at least one of resolution neural networks.

The neural network may include a bypass connection from the pooling layer to the unpooling layer.

Furthermore, a method for replacing missing image data according to another embodiment of the present invention includes: receiving input image data for at least two or more domains among multiple preset domains and information on a target domain; and reconstructing the missing image data of the target domain using a neural network to which the two or more input image data and information on the target domain are input.

The neural network combines the fake image data of a first target domain generated by inputting at least two or more real image data from among the multiple domains as an input, and the real image data, and receives the combined image data as an input to restore the image data. The image and the real image data can be trained using a multi-cycle coherence loss that must be similar.

An apparatus for replacing missing image data according to an embodiment of the present invention includes: a receiver configured to receive input image data for at least two or more domains among multiple preset domains; and a replacement unit for reconstructing missing image data of a preset target domain using a neural network receiving the two or more input image data as inputs.

The neural network combines the fake image data of a first target domain generated by inputting at least two or more real image data from among the multiple domains as an input, and the real image data, and receives the combined image data as an input to restore the image data. The image and the real image data can be trained using a multi-cycle coherence loss that must be similar.

The receiver may receive input image data for the at least two or more domains and information on the target domain together.

The neural network includes a generative adversarial network (GAN), a convolutional neural network, a neural network based on a convolution framelet, and a pooling layer and an unpooling layer. It may include at least one of resolution neural networks.

The neural network may include a bypass connection from the pooling layer to the unpooling layer.

A method of replacing missing image data according to another embodiment of the present invention includes: receiving input image data for at least two or more domains among multiple preset domains; and reconstructing missing image data of a preset target domain corresponding to the two or more input image data using a neural network trained by a predefined multi-cycle coherence loss.

According to embodiments of the present invention, restoration performance may be improved by reconstructing missing image data of a target domain using a neural network using image data of each of multiple domains as an input.

According to embodiments of the present invention, the data acquisition method currently used in cancer diagnosis in the medical field is used without modification, and the problem of missing data that may occur at this time can be replaced without additional cost and imaging. It can significantly save time and money for both parties.

According to embodiments of the present invention, when a missingness occurs in an image set of various contrasts necessary for cancer diagnosis, it may be used to replace missing data, and may be used to replace missing data in various illumination direction data sets, It can also be used to replace missing data in human face data of various expressions. Furthermore, the present invention relates to data missing from various camera angle data, data missing from data according to the resolution of the image, data missing from data according to the noise level of the image, and data missing from data according to the artistic style or type of image. It can be used universally for missing image data generated when various domains exist, such as missing data in data and font type data of characters.

1 shows an exemplary diagram of an image conversion task according to the prior art and the present invention.
2 shows an exemplary diagram for explaining a process of training a neural network in the present invention.
3 is a flowchart illustrating a method for replacing missing image data according to an embodiment of the present invention.
4 shows an exemplary diagram of an MR contrast replacement result.
5 shows the configuration of an apparatus for replacing missing image data according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be embodied in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” refers to a referenced component, step, operation and/or element of one or more other components, steps, operations and/or elements. The presence or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

Embodiments of the present invention make it a gist of reconstructing missing image data of a target domain using a neural network that uses image data of each of multiple domains as an input.

Here, in the present invention, fake image data of a target domain and input image data generated from input image data of multiple domains are combined, and an image reconstructed from the combined image data of multiple domains and original input image data must be similar to multiple cycles. By training the neural network using the coherence loss, a learning model may be generated, and missing image data of the target domain may be reconstructed using the neural network of the generated learning model.

The neural network in the present invention includes a generative adversarial network (GAN), a convolutional neural network, a convolution framelet-based neural network, a pooling layer and an unpooling layer. It may include various types of neural networks such as multi-resolution neural networks, for example, U-Net, etc., and may include all types of neural networks that can be used in the present invention as well. In this case, the multi-resolution neural network may include a bypass connection from the pooling layer to the unpooling layer.

The present invention describes a Collaborative Generative Adversarial Network (CollaGAN) framework that processes multiple inputs to generate more realistic and feasible outputs. The CollaGAN framework of the present invention processes multiple inputs from multiple domains, compared to Star-GAN, which processes a single input and a single output, as shown in Fig. 1c. The image replacement technique of the present invention provides many advantages over existing methods.

First, the basic imaging manifold can synergize with multiple input data sets that share the same manifold structure rather than a single input. Therefore, the estimation of missing values using CollaGAN is more accurate.

Second, CollaGAN still maintains the first-generation architecture similar to StarGAN, which is more memory efficient than CycleGAN.

Hereinafter, the present invention will be described in detail.

Imputation using multiple inputs

으로부터 공동 매핑(collaborative mapping)을 통하여 생성자를 트레이닝시키고 타겟 도메인

의 출력 영상을 합성한다. 여기서, C는 상보 셋(complementary set)을 의미할 수 있다. 이 매핑은 아래 <수학식 1>과 같이 나타낼 수 있다.For convenience of description, it is assumed that there are four types (N=4) of domains a, b, c, and d. The present invention provides a set of different types of multiple images in order to process multiple inputs using a single generator.

Train the constructor through collaborative mapping from the target domain

Composite the output image of Here, C may mean a complementary set. This mapping can be expressed as in <Equation 1> below.

[Equation 1]

는 적절한 타겟 도메인인

에 대한 출력을 생성하도록 가이드하는 타겟 도메인 지수를 의미할 수 있다.here,

is the appropriate target domain.

It may mean a target domain index that guides generating an output for .

Since the number of combinations for multiple input and single output combinations is N, the present invention randomly selects these combinations during training so that the generator can learn various mappings for multiple target domains.

network loss

라고 가정하면, 도 2에 도시된 바와 같이 생성자의 백워드 흐름(backward flow)에 대한 다른 입력으로서 N-1개의 새로운 조합들을 생성할 수 있다. 예를 들어, N = 4인 경우 아래 <수학식 2>와 같이 다중 입력과 단일 출력의 세 가지 조합이 있어, 생성자의 백워드 흐름을 이용하여 오리지널 도메인의 세가지 영상을 재구성할 수 있다.Multi-cycle coherence loss: One of the key concepts of the method according to an embodiment of the present invention is cycle coherence for multiple inputs. Since the input is multiple images, we need to redefine the cycle loss. The output of the forward constructor G

, it is possible to generate N-1 new combinations as another input to the backward flow of the generator as shown in FIG. 2 . For example, when N = 4, there are three combinations of multiple inputs and single outputs as shown in Equation 2 below, so that three images of the original domain can be reconstructed using the backward flow of the generator.

[Equation 2]

Here, the associated multi-cycle coherence loss can be expressed as in Equation 3 below.

[Equation 3]

은 l₁―norm을 의미할 수 있다.here,

may mean l ₁ -norm.

Multi-cycle coherence loss combines the fake image data of the target domain and the input image data generated from input image data of multiple domains, and the loss in which the image restored from the combined multi-domain image data and the original input image data must be similar. can mean

의 사이클 일관성 손실은 아래 <수학식 4> 및 <수학식 5>와 같이 나타낼 수 있다.In general, forward generators

The cycle coherence loss of can be expressed as <Equation 4> and <Equation 5> below.

[Equation 4]

[Equation 5]

Distinguishing Loss : Distinguishers perform two roles, one is to classify the source whether it is real or fake, and the other is to classify the domain types of classes a, b, c, and d. Thus, the discriminator loss can consist of two parts. As shown in FIG. 2 , the discriminator loss can be realized using a discriminator having two paths, D _gan and D _clsf , which share the same neural network weights except for the last layers.

In particular, hostile losses are necessary to make the generated image as real as possible. Regular GAN loss can cause gradient problems that dissipate during the learning process. In order to overcome this problem and improve the robustness of training, the present invention provides Least Square GAN (X. Mao, Q. Li, H. Xie, RY Lau, Z. Wang, and SP Smolley. Least) instead of the original GAN loss. The adversarial loss of squares generative adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2813-2821. IEEE, 2017.) can be exploited. In particular, the discriminator D _gan may be optimized by minimizing the loss in Equation 6 below, and the generator may be optimized by minimizing the loss in Equation 7 below.

[Equation 6]

[Equation 7]

는 상기 수학식 5를 통해 정의될 수 있다.here,

can be defined through Equation 5 above.

과

의 두 부분으로 구성되며, 그것들은 각각 진짜 영상과 가짜 영상의 도메인 분류를 위한 교차 엔트로피 손실일 수 있다. 생성자 G를 트레이닝하는 목적은 타겟 도메인으로 적절하게 분류된 영상을 생성하는 것이다. 그러므로, 본 발명은 먼저 생성자를 적절하게 가이드할 수 있도록 진짜 데이터로만 트레이닝되는 최고의 구분자(classifier) D_clsf가 필요하다. 따라서, 본 발명은 구분자 D_clsf를 트레이닝시키기 위해 손실

를 최소화하며, 그리고 나서 생성자가 정확하게 분류된 샘플을 생성하도록 트레이닝되기 위하여, D_clsf를 고정하면서 생성자 G를 트레이닝함으로써,

를 최소화한다.Next, the domain classification loss is

class

It consists of two parts, which can be cross-entropy loss for domain classification of real and fake images, respectively. The purpose of training generator G is to generate an image properly classified by the target domain. Therefore, the present invention first needs the best classifier D _clsf that is trained only with real data to properly guide the constructor. Therefore, the present invention loses to train the delimiter D _clsf .

By training generator G while fixing D _clsf so that the generator is trained to produce correctly classified samples,

to minimize

를 최소화해야 한다.Specifically, in order to optimize D _clsf , for D _clsf as shown in Equation 8 below,

should be minimized

[Equation 8]

는 진짜 입력

를 클래스

로 정확하게 분류할 수 있는 확률을 의미할 수 있다.here,

is the real input

class

It can mean the probability of being able to accurately classify

On the other hand, generator G must be trained to generate spurious samples properly classified by D _clsf . Therefore, it is necessary to minimize the loss expressed as in Equation 9 below for the generator G.

[Equation 9]

Loss of Structural Similarity Index : The Structural Similarity Index (SSIM) is one of the most advanced indices for measuring image quality. It has been reported in the prior art that l ₂ loss, which is widely used in image restoration tasks, causes blurring artifacts in the result. SSIM is one of the perceptual metrics and can be differentiated, so it can be backpropagated. The SSIM for the pixel p may be expressed as in Equation 10 below.

[Equation 10]

Here, μX means the mean X, σ ² _X means the variance of X, σ _XX* means the covariance of X and X*, and C ₁ and C ₂ are variables for stabilizing the partition, C ₁ = ( k ₁ L ) ² and C ₂ = ( k ₂ L ) ² , where L denotes a dynamic range of pixel intensity, and k ₁ and k ₂ can be 0.01 and 0.03.

Since SSIM is defined between 0 and 1, the loss function for SSIM can be expressed as in Equation 11 below.

[Equation 11]

Here, P may mean a set of pixel positions, and |P| may mean cardinality of P.

The SSIM loss may be applied as an additional multiple cycle consistency loss as shown in Equation 12 below.

[Equation 12]

Mask Vector

To use a single constructor, you need to add a target label in the form of a mask vector to guide the constructor. A mask vector is a binary matrix with the same dimension as the input image, and is easily connected to the input image. The mask vector has a channel dimension of class N that can represent a target domain as a one-hot vector along the channel dimension. This could be a simplified version of the mask vector introduced in the original StarGAN. That is, the mask vector may be target domain information for missing image data to be restored or replaced by using multi-domain input image data input to the neural network.

data set

MR contrast synthesis: A total of 280 axial brain images can be scanned by a multi-dynamic multi-echo sequence and an additional T2 FLAIR sequence from 10 subjects. The data set includes four MR contrast image types, e.g., T1-FLAIR (T1F), T2-weighted (T2w), T2-FLAIR (T2F), and T2-FLAIR* (T2F*). can do. At this time, the three MR contrast image types of T1-FLAIR (T1F), T2-weighted (T2w) and T2-FLAIR (T2F) can be obtained from MAGnetic Venocation image Compilation, and the MR contrast of T2-FLAIR* The figure image type may be obtained by an additional scan with a different MR scan parameter of the third contrast image type (T2F). Details of the MR acquisition parameters can be found in Supplementary Data.

CMU Multi-PIE : You can use a subset of the Carnegie Mellon University Multi-Pose Illumination and Expression Face Database for illumination conversion tasks. The dataset can be selected with five lighting conditions: -90 degrees (right), -45 degrees, 0 degrees (front), 45 degrees and 90 degrees (left) toward the frontal direction of the usual (neutral) facial expressions of 250 participants. . The image can be cut into a screen of a certain pixel size with the face positioned in the center.

Radboud Faces Database (RaFD): RaFD can include 8 different facial expressions, eg neutral, anger, contempt, disgust, fear, happy, sad, and surprise collected from 67 participants. There are three different gaze directions, so a total of 1,608 images can be divided by subjects for training, validation and test sets.

network implementation

The present invention includes two networks, a generator G and a discriminator D, as shown in FIG. In order to obtain the best performance for each task, the present invention can redesign the constructor and the discriminator to suit the properties of each task.

The generator is based on the U-net structure and consists of an encoder part and a decoder part, and each part between the encoder and the decoder is connected by a contracting path. The constructor follows the Net structure, and instead of a batch normalization layer that performs normalization operations and a rectified linear unit (ReLU) layer that performs nonlinear function operations, instance normalization is performed. ) layer (D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.) and Leaky-ReLU layer (K. He, X. Zhang, S (. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.) can be used, respectively.

MR Contrast Conversion: There are various MR contrasts such as T1 weight contrast and T2 weight contrast. A specific MR contrast scan is determined by MRI scan parameters such as repetition time (TR) and echo time (TE). The pixel intensity of an MR contrast image is determined by the physical properties of the tissue, called MR parameters of the tissue, such as T1, T2, and proton density. The MR parameter has a voxel-wise property. This means that for convolutional neural networks, pixel-wise processing is just as important as processing information from neighborhood or field of view (FOV). Therefore, instead of using a single convolution, the generator can use two convolution branches with 1 × 1 and 3 × 3 filters that can handle multi-scale feature information. The two convolutional branches are connected similarly to the inception network.

Lighting transformation: For the lighting transformation task, we can use the original U-Net structure with an instance normalization layer instead of a batch normalization layer.

Facial expression conversion: A plurality of face images with various expressions are input for the facial expression conversion task. The image is not strictly aligned according to the pixel direction, as there is a subject's head movement between the facial expressions. If the original U-net is used for the task between facial expression images, the performance of the creator is deteriorated because information of various facial expressions is mixed in the initial stage of the network. For reference, the features of facial expressions must be mixed in an intermediate stage of the constructor either calculating the features from a large FOV or downsampling them into a pulling layer already. Therefore, the generator can be redesigned with 8 encoder branches for every 8 facial expressions, and connected after the encoding process in the constructor intermediate step. The structure of the decoder is similar to the decoder part of U-net except that more convolutional layers are added using residual blocks.

A discriminator can generally consist of a series of convolution layers and Leaky-ReLU layers. As shown in Fig. 2, the identifier has two output headers, one may be a real or fake classification header, and the other may be a classification header for a domain. The discriminator can use PatchGAN to classify whether a local image patch is real or fake. Dropout is very effective to prevent overfitting of the discriminator. Exceptionally, the discriminator of the MR contrast transform has a branch for multi-scale-processing.

Of course, the neural network in the present invention is not limited to the above-described neural network, and may include all types of networks to which the present invention can be applied. For example, the neural network of the present invention is a generative adversarial network (GAN), a convolutional neural network, a convolution framelet-based neural network, a pooling layer and unpooling ) layer may include various types of neural networks such as, for example, U-Net, and the multi-resolution neural network may include a bypass connection from a pooling layer to an unpooling layer. .

Network training

All models can be optimized using Adam with a learning rate of 0.00001, β1 = 0.9, β2 = 0.999. As mentioned above, the performance of the delimiter should only be linked to genuine labels, which means that it should only be trained using real data. Therefore, the present invention first trains a classifier for a real image with a corresponding label for 10 epochs, and then trains a generator and a classifier. The MR contrast conversion task, lighting conversion, and facial expression conversion task can take about 6 hours, 12 hours, and 1 day, respectively, using the NVIDIA GTX 1080 GPU. YCbCr color code can be used instead of RGB color code for lighting conversion task, and YCbCr coding can consist of Y-luminance and CbCr-color. There are 5 different illumination images, the 3 different illumination images almost share CbCr coding, the only difference is the Y-luminance channel. Therefore, only the Y-luminance channel is processed for the lighting conversion task, and the reconstructed image can be applied to the RGB coded image. The present invention uses RGB channels for the facial expression conversion task, and the MR contrast data set may be composed of a single channel image.

The method according to an embodiment of the present invention trains a discriminator network and a generator network using multi-cycle coherence loss, and when a learning model of the generator network is generated through this training process, the generated generator network, for example, CollaGAN is used. Thus, the missing image data can be replaced or restored. That is, in the method according to an embodiment of the present invention, input image data of multiple domains and information about a target domain, for example, a mask vector, are input to a neural network of a learning model generated through a training process using multi-cycle coherence loss. received, and the missing image data for the target domain may be reconstructed using the learning model of the neural network. The method of the present invention will be described with reference to FIG. 3 as follows.

3 is a flowchart illustrating an operation of a method for replacing missing image data according to an embodiment of the present invention, and may include all of the above-described contents.

Referring to FIG. 3 , in the method for replacing missing image data according to an embodiment of the present invention, input image data for at least two or more domains among preset multi-domains is received ( S310 ).

Here, in step S310, the neural network for replacing the missing image data of the MR contrast image reconstructs the missing image data of at least one of the remaining two target domains using input image data of two of the four domains. When trained, it is possible to receive input image data for two domains, and a neural network for replacing the missing image data of the MR contrast image uses input image data of three of the four domains to form the other target. When trained to reconstruct missing image data for a domain, input image data for three domains may be received. Of course, in step S310, even when it is desired to restore missing image data for a lighting image or a facial expression image, input image data for at least two or more domains may be received through a training process for the input image, and the training process The input of the neural network learned in advance through ? may be determined by a business operator or an individual providing the technology of the present invention.

Furthermore, in step S310, not only input image data for two or more domains but also information on a target domain to be restored, for example, a mask vector may be received together.

When input image data for at least two or more domains is received by step S310, missing image data of a preset target domain is restored using a neural network that receives the received input image data for two or more domains as an input. (S320).

Here, the neural network of step S320 receives information on the target domain as an input, and receives the received missing image data for the target domain based on the input image data for two or more domains and the learning model of the neural network. It can be reconstructed, and the neural network is trained using multi-cycle coherence loss, as described above, so that a learning model can be created.

The neural network in step S320 may be the generator network trained in FIG. 2 , and as described above, the neural network is a generative adversarial network (GAN), a convolutional neural network, and a convolutional framelet (convolution). framelet)-based neural network, a multi-resolution neural network including a pooling layer and an unpooling layer, for example, may include various types of neural networks such as U-Net, and a multi-resolution neural network may include bypass connection from the pooling layer to the unpooling layer, and may include all types of neural networks to which the present invention can be applied.

For example, in the present invention, four domains A, B, C, and D are defined as the entire image data set, and when data called D is missing, data of domains A, B, and C are input to a neural network, for example, a generator network. is used to restore the D image. In the case of a reconstructed image D (fake image), a generator network is trained with the goal of being identified as a real image for the discriminator network to discriminate, and the discriminator network is trained in a direction to distinguish a fake image from a real image. and the generator network proceeds training in a direction to deceive the corresponding discriminator network. Finally, the trained generator network is trained to provide a very realistic and realistic image, so that it is possible to reconstruct missing image data of a desired target domain by inputting multi-domain input image data.

As described above, the method according to an embodiment of the present invention defines the entire image data set for the entire domain, and replaces or restores the image data of the desired target domain by using the existing image data of multiple domains as input to the neural network. can do.

This method according to an embodiment of the present invention uses a neural network to solve the problem of missing data, and it is possible to receive multiple image inputs for the purpose of many-to-one image conversion, and in this process, multiple images are used for stable training. Use cycle coherence loss.

When missing image data is restored using the method according to an embodiment of the present invention, it is possible to restore with much superior performance compared to other algorithms that restore using a single input image. For example, as shown in FIG. 4 , it can be seen that the performance of CycleGAN and StarGAN using only one input in the MR contrast image data set is poor, and the method according to the embodiment of the present invention (proposed) has the performance It can be confirmed that this is excellent.

5 is a diagram illustrating a configuration of an apparatus for replacing missing image data according to an embodiment of the present invention, and shows a conceptual configuration of an apparatus performing the method of FIGS. 1 to 4 .

Referring to FIG. 5 , an apparatus 500 according to an embodiment of the present invention includes a receiving unit 510 and a replacement unit 520 .

The receiving unit 510 receives input image data for at least two or more domains among preset multi-domains.

Here, the receiver 510 receives the missing image data for at least one of the remaining two target domains by using the input image data of two of the four domains for the neural network to replace the missing image data of the MR contrast image. When trained to reconstruct, it is possible to receive input image data for two domains, and a neural network for replacing the missing image data of the MR contrast image uses the input image data of three of the four domains to When trained to reconstruct missing image data for target domains, input image data for three domains may be received.

Furthermore, the receiver 510 may receive not only input image data for two or more domains but also information on a target domain to be restored, for example, a mask vector.

When input image data for at least two or more domains is received, the replacement unit 520 replaces missing image data of a preset target domain using a neural network that receives input image data for two or more domains as an input. restore

Here, the replacement unit 520 receives information on the target domain as an input, and receives the received missing image data for the target domain based on the input image data for two or more domains and the learning model of the neural network. can be restored

In this case, as described above, the neural network is trained using multi-cycle coherence loss, so that a learning model can be generated, and the neural network is a generative adversarial network (GAN), a convolutional neural network, and a convolutional network. It may include a neural network based on a convolution framelet, a multi-resolution neural network including a pooling layer and an unpooling layer, and the multi-resolution neural network is from a pooling layer to an unpooling layer. It may include a bypass connection.

Although the description of the device of FIG. 5 is omitted, the device of FIG. 5 may include all of the contents described with reference to FIGS. 1 to 4 , which is apparent to those skilled in the art.

The system or device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, systems, devices, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA). ), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (13)

receiving input image data for each of at least two or more domains among multiple domains preset in a receiver; and
reconstructing missing image data of a preset target domain by using a neural network to which input image data for each of the at least two or more domains is input by the replacement unit
includes,
The receiving step
Receive input image data for each of the at least two or more domains and information on the target domain together,
The neural network is
Multiple inputs are processed using a single generator, and fake image data of a first target domain generated by using real image data of at least two or more of the multiple domains as input to a generator is combined with the real image data, and the combination The method for replacing missing image data, characterized in that the reconstructed image and the real image data are trained using the multi-cycle coherence loss that is similar to the original image data by using the image data as an input of the generator.
delete delete According to claim 1,
The neural network is
Generative Adversarial Networks (GANs), convolutional neural networks, convolution framelet-based neural networks, and multi-resolution neural networks including pooling and unpooling layers Missing image data replacement method comprising at least one.
delete delete delete a receiver configured to receive input image data for each of at least two or more domains among preset multi-domains; and
A replacement unit for reconstructing missing image data of a preset target domain using a neural network to which input image data for each of the at least two or more domains is input
includes,
the receiving unit
Receive input image data for each of the at least two or more domains and information on the target domain together,
The neural network is
Multiple inputs are processed using a single generator, and fake image data of a first target domain generated by receiving real image data of at least two or more of the multiple domains as input to a generator and the real image data are combined, and the combination The apparatus for replacing missing image data, characterized in that the reconstructed image and the real image data are trained by using the multi-cycle coherence loss that is similar to the original image data by using the image data as an input of the generator.
delete delete 9. The method of claim 8,
The neural network is
Generative Adversarial Networks (GANs), convolutional neural networks, convolution framelet-based neural networks, and multi-resolution neural networks including pooling and unpooling layers Missing image data replacement device comprising at least one.
delete receiving input image data for each of at least two or more domains among multiple domains preset in a receiver; and
Reconstructing missing image data of a preset target domain corresponding to input image data of each of the at least two domains using a neural network learned by a multi-cycle coherence loss predefined in the replacement unit
includes,
The receiving step
Receive input image data for each of the at least two or more domains and information on the target domain together,
The neural network is
Multiple inputs are processed using a single generator, and fake image data of a first target domain generated by using real image data of at least two or more of the multiple domains as input to a generator is combined with the real image data, and the combination The method for replacing missing image data, characterized in that the reconstructed image and the real image data are trained using the multi-cycle coherence loss that is similar to the original image data by using the image data as an input of the generator.

Images