KR102641358B1 - Apparatus and method for domain adaptation of image - Google Patents

Abstract

The present invention relates to an image domain adaptation apparatus and method, and specifically to a technique for modifying the domain of an image. An image domain adaptation apparatus according to an embodiment of the present invention includes an encoder that extracts features (first features) from an image of a source domain and extracts features (second features) from an image of a target domain; a separator that separates style features among the first features and content features among the second features; and an image generator that reconstructs an image of the target domain into an image of the source domain based on style features and content features.

Description

Image domain adaptation apparatus and method {APPARATUS AND METHOD FOR DOMAIN ADAPTATION OF IMAGE}

The present invention relates to an image domain adaptation apparatus and method, and specifically to a technique for modifying the domain of an image.

In domain adaptation of images, conventionally, images are generated relying only on adversarial networks without considering image information (image representation), so the number of encoders and decoders must be increased according to the number of domains, and the number of encoders and decoders for one image must be increased. Since only one output image could be obtained, there were limitations in applying it to images from multiple domains.

Open Paper 1: CyCADA: Cycle-consistent adversarial domain adaptation (International Conference on Machine Learning 2018)

Open Paper 2: Light-weight Calibrator: a separable component for unsupervised domain adaptation (IEEE Computer Vision and Pattern Recognition 2020)

The purpose of the present invention is to provide a domain adaptation technique for images.

Additionally, the purpose of the present invention is to provide a technology for generating images in multiple domains for one image.

Additionally, an object of the present invention is to provide a domain adaptation technique using one encoder and decoder pair.

According to one aspect of the present invention, an encoder extracts features (first features) from an image of a source domain and extracts features (second features) from an image of a target domain; a separator that separates style features among the first features and content features among the second features; and an image generator that reconstructs an image of the target domain into an image of the source domain based on style features and content features.

In one embodiment, the separator may separate features extracted from the image into content features or style features using a non-linear separator.

In one embodiment, the image generator converts content features among the second features into content features in the source domain, and converts the image of the target domain into the source domain using the converted content features and style features among the first features. It can be reconstructed into an image of

According to another aspect of the invention, extracting features (first features) from an image of a source domain; separating style features from the first features; extracting features (second features) from an image of a target domain; separating content features from the second features; An image domain adaptation method is provided, including reconstructing an image of a target domain into an image of a source domain based on style features among first features and content features among second features.

According to another aspect of the present invention, a memory storing instructions for image domain adaptation; and a processor that performs image domain adaptation according to instructions stored in a memory, wherein the instructions extract features from each of a plurality of images having different domains and separate the features from each of the plurality of images into content features and style features. , An image domain adaptation device is provided, which is a command for reconstructing the first image into an image in the domain of the second image based on the content features of the first image among the plurality of images and the style features of the second image among the plurality of images. do.

According to one aspect of the present invention, domain adaptation of images is possible.

Additionally, according to another aspect of the present invention, it is possible to generate images in multiple domains for one image.

Additionally, according to another aspect of the present invention, domain adaptation technology using one encoder and decoder pair is possible.

1 is a diagram for explaining image domain adaptation according to an embodiment of the present invention.
Figure 2 is a block diagram of an image domain adaptation device according to an embodiment of the present invention.
Figure 3 is a flowchart of an image domain adaptation method according to an embodiment of the present invention.
Figure 4 is a diagram showing an example of image domain adaptation according to an embodiment of the present invention.
Figure 5 is a block diagram of an image domain adaptation device according to another embodiment of the present invention.

Hereinafter, various embodiments of this document are described with reference to the attached drawings.

The embodiments and terms used herein are not intended to limit the technology described in this document to a specific embodiment, and should be understood to include various changes, equivalents, and/or substitutes for the embodiments.

In the following description of various embodiments, if a detailed description of a related known function or configuration is judged to unnecessarily obscure the gist of the invention, the detailed description will be omitted.

The terms described below are terms defined in consideration of functions in various embodiments, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

In connection with the description of the drawings, similar reference numbers may be used for similar components.

Singular expressions may include plural expressions, unless the context clearly indicates otherwise.

In this document, expressions such as “A or B” or “at least one of A and/or B” may include all possible combinations of the items listed together.

Expressions such as “first,” “second,” “first,” or “second,” can modify the corresponding components regardless of order or importance and are used to distinguish one component from another. It is only used and does not limit the corresponding components.

When a component (e.g. a first) component is said to be "connected (functionally or communicatively)" or "connected" to another (e.g. a second) component, it means that the component is connected to the other component. It may be connected directly to a component or may be connected through another component (e.g., a third component).

In this specification, “configured to” means “suitable for,” “having the ability to,” or “changed to,” depending on the situation, for example, in terms of hardware or software. ," can be used interchangeably with "made to," "capable of," or "designed to."

In some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components.

For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

Additionally, the term 'or' means 'inclusive or' rather than 'exclusive or'.

That is, unless otherwise stated or clear from the context, the expression 'x uses a or b' means any of the natural inclusive permutations.

Terms such as '..unit' and '..unit' used hereinafter refer to a unit that processes at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software.

1 is a diagram for explaining image domain adaptation according to an embodiment of the present invention.

Referring to Figure 1, image domain adaptation is shown to transform the domain between a source image and a target image according to one embodiment of the present invention. Encoder (E) is a source image (Source image, ) and target image (Target image, ) can be input. The encoder (E) uses the source image ( ) from features (hereinafter referred to as source features) (source feature, ) can be extracted. The encoder (E) uses the target image ( ) from the features (hereinafter, target features) (target feature, ) can be extracted. Separator (S) separates the source features (F _x ) into content features (hereinafter referred to as source content features, ) and style features (hereinafter referred to as source style features, ) can be separated. Separator (S) divides the target features (F _Y ) into content features (hereinafter referred to as target content features, ) and style features (hereinafter referred to as target style features, ) can be separated. At this time, the source content characteristics ( ) is the weight for conversion to the domain of the target image ( ) based, it can be converted to content in the domain of the target image. Additionally, target content characteristics ( ) is the weight ( ) based, it can be converted to content in the domain of the target image. The image generator (G) can reconstruct the image based on the features. That is, the image generator (G) uses the source features ( ) source image reconstructed based on ( ) can be created. The image generator (G) converts the source style features into the domain of the target image ( ) and target style features ( ) and converted to the domain of the target image by reconstructing the source image ( ) can be created. The image generator (G) generates target content features converted to the domain of the source image ( ) and source style features ( ) and converted to the domain of the source image by reconstructing the target image ( ) can be created. The image generator (G) generates target features ( ) The target image reconstructed based on ( ) can be created.

Therefore, according to the invention, the source features ( ), target features ( ), source content features ( ), source style features ( ), target content characteristics ( ) and target style features ( ), etc. are not lost, so domain adaptation is possible by reconstructing the original image or converting the source image to the domain of the target image and the target image to the domain of the source image.

Figure 2 is a block diagram of an image domain adaptation device according to an embodiment of the present invention.

Referring to FIG. 2, the image domain adaptation device 2000 according to an embodiment of the present invention includes an input unit 2100, an encoder 2200, a separator 2300, an image generator 2400, and an output unit 2500. It can be included.

The input unit 2100 can receive an image for image domain adaptation, which converts the domain of the image.

In one embodiment, the input unit 2100 may receive multiple images, and the multiple images may have different domains.

In one embodiment, the input images may be an image of a source domain (source image) and an image of a target domain to be converted to a source domain (target image).

The encoder 2200 can extract features from the image. Specifically, the encoder 2200 may extract features from each of a plurality of images with different domains.

In one embodiment, the encoder 2200 may extract features from each of the source domain image and the target domain image to be converted to the source domain.

In one embodiment, the encoder 2200 may extract features from an image using a model learned based on deep learning, such as a convolutional neural network (CNN).

The separator 2300 may separate features extracted from the image. Specifically, the separator 2300 may separate features extracted from each of a plurality of images with different domains into content features and style features.

In one embodiment, the encoder 2200 may separate features extracted from an image of the source domain into content features (hereinafter, source content features) and style features (hereinafter, source style features). Additionally, the encoder 2200 may separate features extracted from an image of the target domain into content features (hereinafter, target content features) and style features (hereinafter, target style features).

In one embodiment, the encoder 2200 may separate features extracted from the image into content features and style features using a non-linear separator. At this time, the encoder 2200 may separate the features extracted from the image into content features and style features according to Equation 1.

(here, is the content feature in domain A, is a non-linear separation function, are the features in domain A, are style features in domain A)

In one embodiment, the separator 2300 may separate features extracted from an image into content features and style features based on a model learned based on deep learning, such as a convolutional neural network (CNN). .

The image generator 2400 may reconstruct the image based on the features. Specifically, the image generator 2400 reconstructs the image based on the content features of the first image and the style features of the second image among a plurality of images with different domains, thereby converting the first image into the domain of the second image. can be created. At this time, the image generator 2400 converts the content features of the first image into content features in the domain of the second image, and the content features of the first image converted into content features in the domain of the second image. And, the first image converted to the domain of the second image can be generated by reconstructing the image using the style characteristics of the style of the second image.

In one embodiment, the image generator 2400 may generate a target image converted to a source domain based on target content features and source style features. At this time, the image generator 2400 converts the target content features into content features in the source domain and reconstructs the image using the target content features and source style features converted to content features in the source domain. You can create a target image converted to a domain.

Additionally, the image generator 2400 can reconstruct the original image. Specifically, the image generator 2400 may generate a plurality of reconstructed images by reconstructing the image based on features extracted from each of the plurality of images.

In one embodiment, the image generator 2400 may generate a reconstructed image of the source domain area by reconstructing the image based on features extracted from the image of the source domain area.

In one embodiment, the image generator 2400 may generate a reconstructed image of the target domain area by reconstructing the image based on features extracted from the image of the target domain area.

In one embodiment, the image generator 2400 may convert content features of the first domain into content features of the second domain according to Equation 2.

(here, are the features converted from domain A to domain B, is the weight for converting the content features of domain A to domain B, are the content features in domain A, refers to style features in domain B)

The image generator 2400 may reconstruct the image based on a model learned based on deep learning, such as a convolutional neural network (CNN).

The output unit 2500 can output the generated image. Specifically, the output unit 2500 may output an image reconstructed from the original image, an image converted from the first domain to the second domain, and a target image converted to the source domain.

Additionally, the encoder 2200, separator 2300, and image generator 2400 may be composed of a model learned based on deep learning such as a convolutional neural network (CNN).

Additionally, the loss function of the model of the encoder 2200, separator 2300, and image generator 2400 may be determined according to Equation 3 below.

Here, total loss is the reconstruction loss , adversarial loss , consistency loss and perceptual loss It can be composed of:

reconstruction loss Can be determined according to Equation 4 below.

here, silver original image and reconstructed image This may mean a loss for training the encoder (E) and the image generator (G) to minimize the difference between them.

adversarial loss Can be determined according to Equation 5 below.

where, adversarial loss may mean adversarial loss in each of the source domain and target domain.

loss of consistency Can be determined according to Equation 6 below.

Here, X may refer to the source domain and Y may refer to the target domain.

perceptual loss Can be determined according to Equation 7 below.

Here, X may refer to the source domain and Y may refer to the target domain.

Figure 3 is a flowchart of an image domain adaptation method according to an embodiment of the present invention.

Hereinafter, the method shown in FIG. 3 will be described as an example of being performed by the image domain adaptation apparatus 2000 shown in FIG. 2.

In step S3100, an image is input. Specifically, the image domain adaptation apparatus 2000 may receive an image for image domain adaptation to convert the domain of the image.

In one embodiment, the image domain adaptation apparatus 2000 may receive a plurality of images, and the plurality of images may have different domains.

In one embodiment, the input images may be an image of a source domain (source image) and an image of a target domain to be converted to a source domain (target image).

In step S3200, features are extracted from the image. Specifically, the image domain adaptation apparatus 2000 may extract features from each of a plurality of images having different domains.

In one embodiment, the image domain adaptation apparatus 2000 may extract features from each of a source domain image and a target domain image to be converted to the source domain.

In step S33000, the content features and style features are separated. Specifically, the image domain adaptation apparatus 2000 may separate features extracted from each of a plurality of images with different domains into content features and style features.

In one embodiment, the image domain adaptation apparatus 2000 may separate features extracted from an image of the source domain into content features (hereinafter, source content features) and style features (hereinafter, source style features). . Additionally, the image domain adaptation apparatus 2000 may separate features extracted from an image of the target domain into content features (hereinafter, target content features) and style features (hereinafter, target style features).

In one embodiment, the image domain adaptation apparatus 2000 may separate features extracted from an image into content features and style features using a non-linear separator.

In step S3400, the image is reconstructed. Specifically, the image domain adaptation apparatus 2000 reconstructs the image based on the content features of the first image and the style features of the second image among a plurality of images with different domains, thereby converting the first image to the domain of the second image. can be created. At this time, the image domain adaptation apparatus 2000 converts the content features of the first image into content features in the domain of the second image, and converts the content features of the first image into content features in the domain of the second image. The first image converted to the domain of the second image can be generated by reconstructing the image using the style characteristics of the first image and the style of the second image.

In one embodiment, the image domain adaptation apparatus 2000 may generate a target image converted to a source domain based on target content features and source style features. At this time, the image domain adaptation apparatus 2000 converts the target content features into content features in the source domain and reconstructs the image using the target content features and source style features converted to content features in the source domain. A target image converted to the source domain can be created.

Additionally, the image domain adaptation apparatus 2000 can reconstruct the original image. Specifically, the image domain adaptation apparatus 2000 may generate a plurality of reconstructed images by reconstructing the image based on features extracted from each of the plurality of images.

In one embodiment, the image domain adaptation apparatus 2000 may generate an image of the reconstructed source domain area by reconstructing the image based on features extracted from the image of the source domain area.

In one embodiment, the image domain adaptation apparatus 2000 may generate an image of the reconstructed target domain area by reconstructing the image based on features extracted from the image of the target domain area.

In step S3500, the reconstructed image is output. Specifically, the image domain adaptation apparatus 2000 may output an image reconstructed from the original image, an image converted from the first domain to the second domain, and a target image converted to the source domain.

Figure 4 is a diagram showing an example of image domain adaptation according to an embodiment of the present invention.

Referring to Figure 4, the process of converting an image of domain A into an image of domain B is shown. The image domain adaptation apparatus 2000 may extract features from an image of domain A, and separate the extracted features into content features A and style features B. Additionally, the image domain adaptation apparatus 2000 may extract features from an image of domain B and separate the extracted features into content features B and style features B. The image domain adaptation apparatus 2000 can generate an image of domain A converted to domain B by reconstructing the image based on content feature A and style feature B.

Figure 5 is a block diagram of an image domain adaptation device according to another embodiment of the present invention.

As shown in FIG. 5, the image domain adaptation apparatus 2000 includes at least one of a processor 5100, a memory 5200, a storage unit 5300, a user interface input unit 5400, and a user interface output unit 5500. elements, which can communicate with each other via bus 5600. Additionally, the image domain adaptation device 2000 may also include a network interface 5700 for connecting to a network. The processor 5100 may be a CPU or a semiconductor device that executes processing instructions stored in the memory 5200 and/or the storage 5300. The memory 5200 and the storage unit 5300 may include various types of volatile/non-volatile storage media. For example, memory may include ROM 5240 and RAM 5250.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system.

Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Although the embodiments have been described with limited drawings as described above, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

2000: Image domain adaptation device 2100: Input unit
2200: Encoder 2300: Separator
2400: image generation unit 2500: output unit

Claims (15)

an encoder that extracts features (first features) from an image of a source domain and extracts features (second features) from an image of a target domain;
a separator that separates first style features of the source domain among the first features and separates second content features of the target domain among the second features; and
An image generator that reconstructs the image of the target domain into an image of the source domain based on the first style features and the second content features,
The separator separates the first content features of the source domain from the first features by applying the first features or the second features to a non-linear separation function based on a non-linear separator. Separate the second content features from the second features, separate the first style features based on the difference between the first features and the first features applied to the separation function, or separate the first style features from the second features. Separating second style features of the target domain based on differences between the second features applied to the separation function.
delete delete According to paragraph 1,
The image generator,
Converting second content features among the second features into first content features in the source domain, and converting the image of the target domain using the converted content features and first style features among the first features. An image domain adaptation device that reconstructs an image of the source domain.
According to clause 4,
The image generator,
An image domain adaptation device that converts second content features among the second features into first content features in the source domain by the following equation.

(here, are features converted from domain A corresponding to the target domain to domain B corresponding to the source domain, is the weight for converting the content features of domain A to domain B, are the content features in domain A, refers to style features in domain B)
extracting features (first features) from an image of a source domain;
separating style features from the first features;
extracting features (second features) from an image of a target domain;
separating content features from the second features; and
Reconstructing the image of the target domain into an image of the source domain based on style features among the first features and content features among the second features,
The step of separating style features among the first features is
Applying the first features to a non-linear separation function based on a non-linear separator to separate first content features of the source domain from the first features, and applying the first features to the separation function separating the first style features based on differences between the first features applied;
The step of separating content features from among the second features is:
Separate the second content features from the second features by applying the second features to the non-linear separation function, and separate the second content features from the second features based on the difference between the second features and the second features applied to the separation function. A method for image domain adaptation, comprising separating second style features of a target domain.
delete delete According to clause 6,
The step of reconstructing the image of the target domain into the image of the source domain includes:
Converting second content features among the second features into first content features in the source domain, and converting the image of the target domain using the converted content features and first style features among the first features. An image domain adaptation method for reconstructing an image of the source domain.
According to clause 9,
Converting second content features among the second features into first content features in the source domain is performed by the following equation.

(here, are features converted from domain A corresponding to the target domain to domain B corresponding to the source domain, is the weight for converting the content features of domain A to domain B, are the content features in domain A, refers to style features in domain B)
A memory that stores instructions for image domain adaptation; and
A processor that performs image domain adaptation according to instructions stored in the memory,
The command is:
Extract first features from an image of a source domain, extract second features from an image of a target domain, and apply the first features or the second features to a non-linear separation function based on a non-linear separator. Applying to separate first content features of the source domain from the first features or to separate the second content features from the second features, and applying the first features to the separation function Separating the first style features based on the difference between the first style features or separating the second style features of the target domain based on the difference between the second features and the second features applied to the separation function, 2 An image domain adaptation device, which is a command for reconstructing an image of the target domain into an image of the source domain based on content features and the first style features.
delete delete According to clause 11,
Reconstructing the image of the target domain into the image of the source domain,
Converting the second content features into first content features in the source domain and reconstructing the image in the target domain into an image in the source domain using the converted content features and the first style features. Domain adaptation device.
According to clause 14,
Converting the second content features into the first content features is performed by the following equation.

(here, are features converted from domain A corresponding to the target domain to domain B corresponding to the source domain, is the weight for converting the content features of domain A to domain B, are the content features in domain A, refers to style features in domain B)

Images