KR102166016B1 - A domain-aware method for image-to-image translations - Google Patents

Abstract

Disclosed is a domain recognition based conversion technique for image-to-image conversion. According to an embodiment of the present invention, a domain conversion method can comprise the following steps of: separating each of an input image and a target image into a background area and an instance area; inputting the background area and the instance area of each of the separated input image and target image to a conversion model generated for image conversion to train the same; and integrating the converted image obtained through the training by using a label generated based on the instance areas of the separated input image and target image.

Description

A domain-aware conversion technique for image-to-image conversion {A DOMAIN-AWARE METHOD FOR IMAGE-TO-IMAGE TRANSLATIONS}

The description below relates to the image conversion technology.

Generative adversarial networks (GANs) are a kind of artificial neural network, a generator that generates a fake image and a discriminator that distinguishes the real image from the generated image. Train. GANs have a good effect in domain transfer and show state-of-art performance in fields such as text-to-image synthesis, super-resolution, and image-to-image translation.

Research using GANs for medical imaging is being conducted in various ways. Most of the research is focused on synthesis, segmentation, and reconstruction. In addition, most of the studies targeting CT and MR data are being conducted, and although a small percentage is being studied, studies are being conducted on Retinal Fundus Imaging, histopathology, and X-ray.

Image to Image (I2I) is used in various computer vision fields such as image denoising, image synthesis, and resolution enhancement.

A domain-aware generative adversarial network (GAN) framework (domain-aware GAN, DAGAN) that segregates instances of the original image using an identity matrix Suggest.

The domain conversion method includes: separating each of an input image and a target image into a background area and an instance area; Inputting and training a background area and an instance area of each of the separated input image and target image into a transformation model generated for image transformation; And integrating the transformed image obtained through the learning by using a label generated based on an instance area of each of the separated input image and target image.

The separating step includes obtaining an instance area separated from each of the input image and the target image, and padding the instance area with an average of pixels other than the acquired instance area to create a background area. It may include the step of obtaining.

In the integrating step, as the input image and the target image are separated, the position value of each instance area is stored as a unit matrix and stored as a label, and the converted image is integrated using the stored label to transform the image. It may include a step of.

In the training step, a transformation model including a background model and an instance model is generated for the image transformation, and the background regions of the input image and the target image are mapped to a common potential space configured in the transformation model, and the The method may include generating a plurality of instance areas by mapping style codes and content codes shared by each instance area to each instance area of the input image and the target image.

The background model includes performing training for cross-domain transformation by inputting a background region of each of the input image and the target image into a background network, and the instance model is a reconstruction mode. And a multi-output mode, wherein the restoration mode includes an instance area of each of the input image and the target image through an instance network when the background area of each of the input image and the target image is changed. The training is performed so that the integrated image matches more than a preset criterion without changing the value, and the multi-output mode is a random pattern in each instance area as the instance area of each of the input image and the target image is learned by the instance network. By adding code, it is possible to learn to change a plurality of styles to obtain output of multiple images.

The domain conversion system includes: a separation unit that separates each of an input image and a target image into a background area and an instance area; A learning unit that inputs and trains a background region and an instance region of each of the separated input image and target image into a transformation model generated for image transformation; And an integration unit for integrating the transformed image obtained through the learning by using a label generated based on an instance area of each of the separated input image and target image.

The domain conversion system according to an embodiment may separate a background region and an instance region from an original image using a unit matrix, obtain a common potential space using UNIT as a base model, and maintain model stability through this. In addition, it is possible to perform domain transformation of the non-corresponding data set more flexibly through learning.

1 is an example of domain recognition conversion of a non-corresponding image according to an embodiment.
2 is an example showing that the domain conversion system is composed of two parts, a background and an instance model according to an embodiment.
3 is a diagram for describing a background model in a domain conversion system according to an embodiment.
4 and 5 are diagrams for explaining an instance model in a domain conversion system according to an embodiment.
6 is a diagram for explaining converting an image in a domain conversion system according to an exemplary embodiment.
7 is a diagram for explaining a smoothed discriminator in a domain conversion system according to an embodiment.
8 is a block diagram illustrating a configuration of a domain conversion system according to an embodiment.
9 is a flowchart illustrating a method of performing image-to-image conversion in a domain conversion system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is an example showing domain recognition transformation of a non-corresponding image.

The background and instance domains may be converted, respectively, and the converted background domains and instance domains may be integrated through a smooth label. Hereinafter, a method of integrating images will be described in detail.

2 is an example showing that the domain conversion system is composed of two parts, a background and an instance model according to an embodiment.

The model proposed by the domain conversion system can perform instance-aware translation between the input image A and the target image B. At this time, it is assumed that each image is composed of a background area and an instance area as follows.

The transformation model proposed by the domain transformation system can be composed of a background model and an instance model. The domain transformation system can cut the instance regions A _ins and B _ins from each of the original images including the input image A and the target image B, before training the entire network. The domain conversion system may store a position value obtained by cutting an instance region from each original image as a unit matrix and use it as a label. The cropped instance regions can be replaced with the average value of the remaining images. After training the background network and the instance network configured in the transformation model, an integration process of restoring the transformed image may be performed.

3 is a diagram for describing a background model in a domain conversion system according to an embodiment.

The background model is for applying different translations for different parts. Referring to FIG. 3, the background model aims at learning a cross-domain transformation between two background inputs.

Conversion can be performed using the instance domain and background domain as examples. By separating the input image, an instance area may be obtained. The background area may be obtained by padding with an average of pixel values. The background model is independent of the entire framework, and the background model can learn visual transformation between the two domains, including the instance domain and the background domain. Through this inference, an encoder and a generator can be defined as follows, respectively.

를 통해 주어진 목표 백그라운드(target background)

and

일 때, 기본적인 잠재적 공간을 맵핑할 수 있다.

와

는 각각 잠재 코드(latent code)를 나타낸다. 실시예에서는 마지막 두 레이어

와 첫번째 레이어

의 가중치 공유(weight sharing)를 수행할 수 있다. 동시에 두 개의 판별기(discriminator)

를 추가할 수 있다. 판별기들은 두 백그라운드 도메인 사이의 변환이 적절한 것인지를 판별하고

와

의 적재적 프로세스를 수행할 수 있다. 이때, 적대적 프로세스란

와

가

에 의해 판별되는 것을 의미한다. 동시에

와

는

에 의해 판별될 수 있다. Similar to UNIT, encoder

Target background given through

and

When is, basic potential space can be mapped.

Wow

Each represents a latent code. In the example, the last two layers

And the first layer

Weight sharing of may be performed. Two discriminators at the same time

Can be added. The discriminators determine whether the conversion between the two background domains is appropriate

Wow

Can carry out the load-bearing process. At this time, the hostile process

Wow

end

Means to be determined by At the same time

Wow

Is

Can be determined by

4 and 5 are diagrams for explaining an instance model in a domain conversion system according to an embodiment.

The instance network may be configured in reconstruction and multi-output modes. The goal of the instance model is to keep the instance domain independent. In other words, it is not affected by background transitions. First, a simple instance restoration is performed with a structure similar to DCGAN (Deep Convolution Generative Adversarial Networks), and a noise vector following a Gaussian distribution N(0,1) is extracted. This allows the entire framework (including the background network) to transform the background region into a different style, but the instance region remains unchanged. The image is composed of styles and content codes, and the instance part can be converted into a multiple output model.

In the reconstruction mode, the last unified image may be created to be as realistic as possible without changing the instance area. Considering the size of the instance area, a plurality of convolutional layers (eg, 3) inside the generator and discriminator can be placed, and random noise Z ~ N (0, 1) sampled from the normal distribution can be used as an input vector. . Referring to FIG. 4, a reconstruction mode is shown, and a discriminator composed of three blocks can distinguish an input of an original image and a result G(z) generated from an input of the original image. Each block of the generator can consist of "Transposed convolution + Batch normalization + Relu activation". The discriminator block is distinguished from the one composed of "Convolutional + Batch normalization + leakly Relu activation".

Referring to FIG. 5, in the multiple-output mode, all desired conversion operations may be performed on the instance area under the condition that the instance area cut from the original image is completely independent. In a general approach, images are viewed as a combination of style and content information to obtain various outputs. In particular, you can convert an image to an arbitrary style if you add appropriate attributes or style codes. Similar to the structure of multimodal unsupervised image-to-image translation (MUNIT), it is possible to generate multiple instances by randomly sampling the style code obtained from the target instance area and recombining it with the content code.

에서 입력 인스턴스 영역A _ins 와 목표 인스턴스 영역 B _ins를 스타일과 컨텐츠 공간(content space)으로 각각 맵핑할 수 있다(

는 입력 인스턴스 영역A _ins 와 목표 인스턴스 영역 B _ins의 스타일과 컨텐츠 공간을 나타낸다.).Given an input instance area A _ins and a target instance area B _ins , the encoder

In, you can map the input instance area A _ins and the target instance area B _ins to style and content space respectively (

_Represents the style and content space of the input instance area A _ins and the target instance area B _ins ).

및 입력 인스턴스 영역A _ins의 스타일

에서 무작위로 샘플링 된 두 스타일 코드

와 입력 인스턴스 영역A _ins 와 목표 인스턴스 영역 B _ins의 컨텐츠 코드

,

를 통합할 수 있다. 학습 과정에서 획득된 순환적 의미(Cycle sementics)를 통해 양방향 인스턴스 변환을 수행할 수 있다. 도 5는 Pascal VOC 데이터 셋에 다중 출력 모델을 적용한 예이다. Like the common space setting of the background model, the content space can also be shared by the input instance area and the target instance area, that is, two instance areas. After that, the style of the target instance area B _ins

And the style of the input instance area A _ins

Two style codes randomly sampled from

And the content code of the input instance area A _ins and the target instance area B _ins

,

Can be integrated. Bi-directional instance conversion can be performed through cycle sementics acquired in the learning process. 5 is an example of applying a multiple output model to a Pascal VOC data set.

The loss function of the model proposed in the embodiment may be used differently depending on the background model and the instance model. The full object function to solve the problem of background and instance conversion is as follows.

The instance model may be composed of a reconstruction mode and a multi-output mode for flexible transformation. The restoration mode prevents the instance area from disappearing when the background area is transformed, and the multiple output mode may be used to generate transformations of various instance areas. For example, after cropping the instance area from the input image, it is possible to change the style by adding arbitrary code to the instance area.

로 구성되며, 두 개의 생성기와 판별기

로 구성될 수 있다. 이때, 각 생성기와 판별기는 3개의 레이어로 구성될 수 있다. 적대적 손실(adversarial loss)은 아래와 같이 나타난다.4, the instance network in the reconstruction mode is a noise vector

Consists of two generators and a discriminator

It can be composed of. At this time, each generator and discriminator may be composed of three layers. Adversarial loss appears as follows.

으로 구성되며, 다중 출력 모드에서의

은 복원 손실(reconstruction loss)과 적대적 손실(adversarial loss)로 구성될 수 있다. As can be seen in FIG. 5 showing the multiple output mode, the input instance area A _ins is a set of encoder-generators.

And in multiple output mode

May be composed of reconstruction loss and adversarial loss.

The loss function and training will be described.

In general, the restoration loss can be processed by converting the instance area into an image for latent code. However, the style code is not sampled from the normal distribution N(0,1). In the model according to the embodiment, the transformed instance area using the style encoder and the content encoder maintains the properties of the target domain. The goal of this loss function is to allow the image to restore latent code and semantic consistency after learning.

와

를 사용하여 변환된 인스턴스 영역과 실제 인스턴스를 구분할 수 있다. The main purpose of using GAN in hostile loss is that it is possible to obtain a result that matches the target domain as much as possible through the hostile learning process. Discriminator

Wow

You can use to distinguish between the transformed instance area and the actual instance.

The background loss will be described.

로 구성될 수 있다. 백그라운드 모델과 인스턴스 모델은 통합 이전에는 서로 독립적이다. 백그라운드 모델의 시각적 도메인은 복원과 변환 스트림(reconstruction and translation stream)을 따른다. 백그라운드 모델은 VAEs와 GANs를 표현하는

와 의미론적 일관성 손실(semantic-consistency loss)를 사용할 수 있다. 세 개의 가중치 파라미터

는 각각의 컴포넌트의 영향을 측정하는데 적용될 수 있다. The background model is

It can be composed of. Background model and instance model are independent of each other prior to integration. The visual domain of the background model follows a reconstruction and translation stream. The background model expresses VAEs and GANs

And semantic-consistency loss can be used. Three weight parameters

Can be applied to measure the impact of each component.

The VAE structure uses the lower bound of latent codes (ELBO), which aims to learn the latent model through approximation of the marginal log-likelihood of the training dataset, and the objective function is as follows.

는 목적 함수와 쿨백-라이블러 발산(KL divergence) 의 영향력을 조절한다. 쿨백-라이블러 발산은

이 얼마나 사전 확률

와 비슷한가를 추정하는 평가 지표이다. 사전 확률은 공통 컨텐츠 공간(common content space)

의 분포를 나타낸다. 실시예에서는 샘플링의 성능을 높이기 위하여 라플라시안 분포(Laplacian distributions)를 따르는

와 표준 분포를 따르는

를 각각 모델링 할 수 있다.The weight parameter in the objective function

Controls the influence of the objective function and the KL divergence. Coolback-Leibler divergence

What a prior probability

It is an evaluation index that estimates whether it is similar to Prior probabilities are common content space

Shows the distribution of In an embodiment, in order to improve the performance of the sampling, Laplacian distributions are followed.

And follow the standard distribution

Each can be modeled.

The objective function of GAN aims to transform and restore images in hostile processes.

) 을 사용하여, 잠재적 공간의 거리를 측정하는 쿨백-라이블러 발산을 대신하여 바로 의미론적 차이를 비교할 수 있다.The semantic-consistency loss function allows the image to be mapped back to the original potential space, and at this time, the characteristics of the target domain are not lost. The model according to the embodiment is L1 norm (

), we can directly compare the semantic differences in place of the Coolback-Leibler divergence, which measures the distance of the potential space.

6 is a diagram for explaining converting an image in a domain conversion system according to an exemplary embodiment.

The domain conversion system according to an embodiment may divide an image into a background area and an instance area, and then independently input the two separated areas into the conversion network. You can then use the saved label to merge the two transformed regions. At this time, it should be considered that how to make it look real through the integration process, that if you simply cut and then transform and integrate each, the integration result will not be good because the background area and the instance part have completely different directions.

Referring to FIG. 6, as an example showing the transformation when there is no "smoothing", the image is divided into a background area (Bgr) and an instance area (Ins), and the background area and the instance area are model _bgr which is a background network and an instance network. It can be entered as model _ins . After the conversion is complete, the results generated by the two networks can be integrated into the entire image.

로 표현되는 어느 포인트에 대하여 아래와 같이 정리될 수 있다.A label smoothing technique can be used to keep the image realistic after integration. Specifically, GAN is known to be effective when the discriminator can estimate the ratio between the input and model data.

For any point expressed as, it can be summarized as follows.

According to the conditions of the previous step, the discriminator can be learned as follows.

(Instance) restore mode:

(Instance) Multiple output mode:

_Suppose model _bgr represents the background network and model _ins represents the instance network. If the output generated by the model has gone through the integration process, it can be expressed as follows if the entire image I _A and I _B are represented.

를 추가하여 훈련 데이터를 스무딩(smoothing)한다. 파라미터는 아래와 같이 주어질 수 있다.After integration, another pair of discriminators {D _A , D _B } can be added to distinguish B , I _A from A, I _B. Since the integration is not exactly the same as the original domains A and B, the parameters

The training data is smoothed by adding. Parameters can be given as follows.

를 판별 프로세스에 추가하기 위한 한 쌍의 판별기 {D _A , D _B }를 이용하여 비율을 측정할 수 있다. Then, the parameters

The ratio can be measured using a pair of discriminators {D _A , D _B } to add to the discrimination process.

가 원본 이미지와 비교하여 출력을 덜 확신하게(less confident)하고 출력의 스타일에 영향을 주기는 하지만, 이 조절(adjustment)은 통합을 더 실제적으로 만들고, 이미지가 완전히 다른 두 이미지에서 합성된 것을 눈치챌 수 없게 한다.Smoothing parameter

Is less confident in the output compared to the original image and affects the style of the output, but this adjustment makes the integration more realistic, and you notice that the image is composited from two completely different images. Make it impossible to hit.

Referring to FIG. 7, it is a diagram for explaining a smoothed discriminator. The instance area A _ins and the background area A _bgr are separated from the input image A, and the entire image I _A can be generated through the label through the instance model model _ins and the background model model _bgr . The model according to the embodiment may add a new discriminator D _A to distinguish I _A from the target image B.

8 is a block diagram illustrating a configuration of a domain conversion system according to an embodiment, and FIG. 9 is a flowchart illustrating a method of performing image-to-image conversion in a domain conversion system according to an embodiment.

The processor of the domain conversion system 800 may include a separation unit 810, a learning unit 820, and an integration unit 830. Components of such a processor may be expressions of different functions performed by the processor according to a control command provided by a program code stored in the domain conversion system 800. The processor and components of the processor may control the domain conversion system 800 to perform steps 910 to 930 included in the method of performing image-to-image conversion of FIG. 9. In this case, the processor and the components of the processor may be implemented to execute an instruction according to the code of the operating system included in the memory and the code of at least one program.

The processor may load program code stored in a file of a program for a method of performing image-to-image conversion into a memory. For example, when a program is executed in the domain conversion system 800, the processor may control the domain conversion system 800 to load the program code from the program file into the memory under the control of the operating system. At this time, in the processor and the processor included in the processor, each of the separating unit 810, the learning unit 820, and the integrating unit 830 executes a command of a corresponding part of the program code loaded into the memory, and the subsequent steps 910 to 930) may be different functional expressions of the processor.

In step 910, the separating unit 810 may separate each of the input image and the target image into a background area and an instance area. The separating unit 810 may obtain an instance area separated from each of the input image and the target image, and obtain a background area by padding the remaining area other than the obtained instance area with the average of the pixels. .

In step 920, the learning unit 820 may input the separated input image and the background area and the instance area of each of the target image into a transformation model generated for image transformation and train it. The learning unit 820 generates a transformation model including a background model and an instance model for image transformation, maps the background region of each of the input image and the target image to a common potential space configured in the transformation model, and A plurality of instance areas may be created by mapping style codes and content codes shared by each instance area to each instance area of the image. In this case, the background model may perform training for cross-domain transformation by inputting a background region of each of the input image and the target image to the background network. The instance model includes a reconstruction mode and a multi-output mode, and the reconstruction mode includes an input image and an input image through the instance network when the background area of each of the input image and the target image is changed. The training is performed so that the integrated image matches more than a preset criterion without changing the instance area of each target image, and in the multiple output mode, each instance is learned by learning the instance area of each of the input image and the target image to the instance network. By adding an arbitrary code to an area, it is possible to learn to obtain output of multiple images by changing a plurality of styles.

In step 930, the integrating unit 830 may integrate the transformed image acquired through learning by using a label generated based on the separated input image and the instance area of each of the target image. The integration unit 830 may convert the image by storing the position value of each instance area as a unit matrix as a unit matrix and storing it as a label, and integrating the converted image using the stored label as each of the input image and the target image is separated. have.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (6)

In the domain conversion method,
Separating each of the input image and the target image into a background area and an instance area, respectively;
Inputting and training a background area and an instance area of each of the separated input image and target image into a transformation model generated for image transformation; And
Integrating the transformed image obtained through the learning using a label generated based on the separated input image and each instance area of the target image
Including,
The learning step,
For the image transformation, a transformation model including a background model and an instance model is generated, and the background regions of the input image and the target image are mapped to a common potential space configured in the transformation model, and the input image and the target image are each Creating a plurality of instance areas by mapping style codes and content codes shared by each instance area to the instance area of
Domain conversion method comprising a. The method of claim 1,
Separating each of the above,
Acquiring an instance area separated from each of the input image and the target image, and padding the instance area with an average of pixels in the remaining area other than the acquired instance area to obtain a background area
Domain conversion method comprising a. The method of claim 1,
The integrating step,
As the input image and the target image are separated, the position value of each instance area is stored as a unit matrix and stored as a label, and the image is converted by integrating the converted image using the stored label.
Domain conversion method comprising a. delete The method of claim 1,
The background model includes performing training for cross-domain transformation by inputting background regions of each of the input image and target image to a background network,
The instance model includes a reconstruction mode and a multi-output mode,
In the restoration mode, when the background area of each of the input image and the target image is changed, learning is performed so that the integrated image matches a preset criterion or more without changing the instance area of each of the input image and the target image through an instance network. Perform,
The multiple output mode is trained to acquire output of multiple images by changing a plurality of styles by adding an arbitrary code to each instance area as the instance area of each of the input image and the target image is learned by the instance network. Letting
Domain conversion method, characterized in that. In the domain conversion system,
A separating unit for separating each of the input image and the target image into a background area and an instance area;
A learning unit that inputs and trains a background region and an instance region of each of the separated input image and target image into a transformation model generated for image transformation; And
An integration unit that integrates the transformed image obtained through the learning using a label generated based on the separated input image and the instance area of each of the target image
Including,
The learning unit,
For the image transformation, a transformation model including a background model and an instance model is generated, and the background regions of the input image and the target image are mapped to a common potential space configured in the transformation model, and the input image and the target image are each To create multiple instance areas by mapping the style code and content code shared by each instance area to the instance area of
Domain conversion system.

Images