KR102192211B1 - Efficient Generative Adversarial Networks using Depthwise Separable and Channel Attention for Image to Image Translation - Google Patents

Abstract

Provided are an image translation method of efficient generative adversarial networks (GANs) using a depthwise separable convolution and channel attention, and an apparatus thereof. According to the present invention, the image translation method of efficient generative adversarial networks (GANs) using a depthwise separable convolution and channel attention comprises the steps of: applying a depthwise separable convolution to reduce the number of parameters; and applying channel attention to balance the quality of an output image and computational cost by compensating for information loss that occurs by applying the depthwise separable convolution.

Description

Efficient Generative Adversarial Networks using Depthwise Separable and Channel Attention for Image to Image Translation}

The present invention relates to a method and an apparatus for transforming an image through an efficient hostile generated neural network using convolution and channel attention separable by depth.

Recently, studies on image conversion are actively being studied. Image transformation has a variety of forms, from classical methods using geometric methods and filtering to methods using deep learning. In particular, the deep learning method has achieved meaningful results due to the rapid development of artificial neural networks. Among them, the field that has achieved the greatest achievements in image transformation research is research using Generative Adversarial Networks (GANs). Image conversion through a hostile generation neural network converts the style of an image into an image form of another domain (Image-to-Image Translation), creates a new image with high resolution (Super-resolution), or repairs damaged parts of an image. This includes image inpainting and image restoration. It is used in the overall field of the industry, such as improving autonomous driving data sets, restoring and improving medical images, and enhancing class imbalance data sets. However, the hostile generated neural network consists of several existing convolutional neural networks, and thus has a heavier structure and a large amount of computation compared to other artificial neural networks. Therefore, the number of parameters expressing the complexity of the deep learning model appears very high. The high number of parameters not only significantly affects the learning time and the inference time to generate the output, but also increases the memory resources required for model training and inference. These problems become a big limitation in applying the conversion between images to various fields in the real world.

1 is a structure of a basic hostile generated neural network according to the prior art.

Recently, a large amount of research has been conducted on hostile neural networks. The earliest model proceeded in a simple form as shown in FIG. 1.

A generator that receives random noise as an input learns to generate a fake image, that is, an image generated by the generator through random noise. Conversely, the discriminator distinguishes between the fake image created by the creator and the real image of the provided dataset. In this process, the creator learns to create an image that is indistinguishable from the real image in order for the fake image to deceive the discriminator, and the discriminator is learned in the direction to distinguish the fake image from the real image. Since the constructor and the discriminator are in an adversarial relationship, they are contradictory to each other, expressed by the following formula:

As described above, the basic hostile generation neural network consists of two networks. In some cases, it is composed of one basic constructor and one discriminator, but it is composed of several constructors and discriminators depending on the purpose.

Image-to-Image Translation is a research field that changes the style of datasets in different domains. It can be used in a variety of fields, and recently, many studies that produce great results have been shown. However, the domain data set used in the previous study was mainly a paired data set. This has a problem that it is expensive to build a dataset, and that it is difficult to apply it to reality because most of the datasets that exist in the real world are not paired. To solve this problem, research has begun to focus on the use of unpaired datasets, and many achievements have been achieved. In the prior art CycleGAN, a hostile generated neural network model has been disclosed that maps image datasets between two domains, learns cycle consistency loss, and transforms each domain into images in different domain styles. In CycleGAN, since the aforementioned unpaired data set is used, it has a great advantage in building the data set, so studies that solve the problem by applying it in various fields are being shown. Unlike general hostile generated neural networks, it consists of two generators (G, F) and two discriminators (Dx, Dy), which has a relatively high computational amount and requires a relatively large amount of memory resources. Research has been conducted to create images of better quality using the versatility of CycleGAN.

Another prior art AttentionGAN is a hostile generation neural network based on CycleGAN. Like CycleGAN, it is a model that performs conversion between images through mapping between two domains using cyclic coherence loss. In addition, the fact that the attention mechanism was used helped to create better quality images. AttentionGAN extracts feature points from the same network and supplies them to two other networks. The Content Mask Generator creates a Contents mask that focuses on the characteristics of the main object for the input image. The Attention Mask Generator creates an attention mask that learns features of an important part of the dataset. By separating the features of the image in this way, each network performs detailed learning. Content masks and attention masks from each constructor are normalized, and multiplication operation is performed on the matrix to preserve the parts to be focused and erase parts that are not learned from the attention mask. All of these are performed by the built-in function of AttentionGAN, so more efficient learning can be performed. However, both CycleGAN and AttentionGAN have the disadvantage of only 1:1 unimodal output. In the end, there is a problem that it is inefficient because it requires multiple learning to generate multiple style images in one domain. To solve this problem, models that output multimodal were studied. Studies focusing on multimodal are more efficient in outputting different images than the aforementioned unimodal model.

Another prior art StarGAN consists of one constructor and N constructors, where N denotes the number of domains to be created. One creator learns the input image and domain label, and the discriminator of the label learns to distinguish between real data and fake data. This learning is efficient because, unlike the existing Unimodal model, images of various domains can be learned and created with only one constructor.

Unlike StarGAN, another conventional rider MUNIT can create various domain images without a domain label. This is possible because, like AttentionGAN, the feature points are extracted by disentangled, and these are separately injected into the Contents encoder and the Style encoder. MUNIT is likewise composed of one constructor and one discriminator, so more efficient multimodal generation is possible.

As the number of channels of a feature increases, the computations that occur in the deep learning network increase and the cost increases. Research to solve this problem has been continued, and studies based on depth-specific separation convolution have shown good results. Inspired by the prior art Inception and depth-specific convolution, Xception proposed an extreme version starter module that performs 1x1 conversion first and then performs spatial correlation mapping individually for all output channels.

MobileNet uses a depth-separable convolution and an additional contraction hyperparameter consisting of a width multiplier to control the input and output channels and a resolution multiplier to scale the input image. ShuffleNet emphasizes that pointwise convolution is still a high cost domain. To solve this problem, ShuffleNet designed a channel sparse that did not connect all the weights and mixed groups to avoid the problem of getting only the information flow for a specific area as input.

Existing attention mechanisms have been mainly used in the field of natural language processing, but recently, they are also used in various ways in the field of computer vision. It is used in various ways such as Image Classification, Super-Resolution, and Image Detection, and various kinds of attention mechanisms are being studied. Channel Attention used in SeNet analyzes the dependence between channels by compressing and rescaling channel information and improving performance by giving a higher weight to the part to be focused. In addition, CBAM improves performance by additionally scaling spatial information by combining a spatial attention mechanism as well as a channel.

The technical problem to be achieved by the present invention is to replace the general convolution with depthwise separable convolution in unpaired image-to-image translation, and a DCBlock module that applies channel attention. It provides and tries to reduce the number of learning parameters so that it can be used in an environment with limited memory resources through DCBlock.

In one aspect, the image conversion method of an efficient hostile-generated neural network using separable convolution for each depth and channel attention proposed in the present invention includes the steps of applying a depth separable convolution and a depth separable convolution to reduce the number of parameters. And applying a channel attention to balance between the quality of the output image and computational cost by compensating for information loss caused by the application.

The step of applying depth-separable convolution to reduce the number of parameters is to apply convolution by depth using the kernel size, width, height, input channel and feature map of the convolution by depth, and do not deal with spatial features. The point convolution that is performed is applied.

The step of applying channel attention to balance between the quality of the output image and computational cost by compensating for information loss that occurs by applying depth-separable convolution is to focus on the required input feature points by utilizing interdependence between channels, and global average The channel information is compressed using Global Average Pooling (GAP), the input feature points are restored through the convolution layer, and a gating mechanism is used to reinforce and restore the input feature points for the part to be focused in the network. Channel statistics are extracted.

Another image conversion device for an efficient hostile-generated neural network using depth-separable convolution and channel attention proposed in the present invention provides a depth-separable convolution and a depth-separable convolution to reduce the number of parameters. And a channel attention unit for applying channel attention to balance between the quality of the output image and computational cost by compensating for information loss caused by the application.

The DCBlock according to the embodiments of the present invention includes a Depthwise Separable Convolution that has a relatively lower computational cost than a Standard Convolution and an information loss caused by separable convolution by depth ( In spite of the maximum 91.6% reduction in the number of parameters, an index expressing the amount of computation, by applying channel attention to compensate for information loss), an image of similar quality is generated compared to the models of the prior art. Shows the performance.

1 is a structure of a basic hostile generated neural network according to the prior art.
FIG. 2 is a diagram illustrating a structure of an image conversion apparatus for an efficient hostile generated neural network using convolution and channel attention that can be separated by depth according to an embodiment of the present invention.
3 is a flowchart illustrating an image conversion method of an efficient hostile-generated neural network using convolution and channel attention separable for each depth according to an embodiment of the present invention.
4 is a diagram for describing a convolution configuration that can be separated by depth according to an embodiment of the present invention.
5 is a diagram for explaining a channel attention mechanism according to an embodiment of the present invention.

According to an embodiment of the present invention, in order to reduce the number of learning parameters, a general convolution is replaced with a depthwise separable convolution. However, separable convolution for each depth reduces the number of parameters, but causes information loss accordingly. Information loss has a decisive effect on the generation of images of poor quality by the hostile generated neural network model.

In order to compensate for information loss that occurs when a convolution separable by depth is applied to reduce the number of learning parameters, in the present invention, channel attention is applied to a residual block. When channel attention is applied, it focuses on important features and mitigates information loss, thereby ensuring image quality.

Therefore, in the present invention, DCBlock is proposed by applying the aforementioned techniques. DCBlock has a structure that performs image-to-image conversion by replacing the “residual block” used in general hostile neural networks. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 2 is a diagram illustrating a structure of an image conversion apparatus for an efficient hostile generated neural network using convolution and channel attention that can be separated by depth according to an embodiment of the present invention.

In the present invention, a Depthwise separable Channel Attention Block (DCBlock) is proposed in order to solve the problem that the learning model becomes heavy during unpaired image-to-image translation using a hostile generated neural network.

As can be seen in FIG. 2, the proposed module DCBlock includes a depth-specific separation unit 210 (that is, a depth-specific separation session (Depthwise Separable Session)) and a channel attention unit 220 (that is, a channel attention session (Channel Attention Session)).

The depth-separating unit 210 applies a depth-separable convolution to reduce the number of parameters.

Depth-specific separation unit 210 is a Dethwise Separable Conv block, Ins Norm (Instance Normalization) block, ReLU block, Dethwise Separable Conv block, Ins Norm (Instance Normalization) block Through this, convolution for each depth is applied using the kernel size, width, height, input channel and feature map of the depth convolution, and point convolution that is performed only in the channel is applied without dealing with spatial features.

Information loss occurs when the number of parameters is reduced by applying depth-separable convolution. This loss of information can be overcome with residual linkage, but the result of the model is not as good as the image quality of the existing model because it is not alleviated in the conversion between images. Information loss occurs because deeply separable convolution is performed for each channel, and loss of features appearing in the entire portion is inevitable. Therefore, it was necessary to add a technique to compensate for information loss in order to achieve an optimal balance between the quality of the output image and the computational cost. Therefore, channel attention is applied so that necessary information is kept as much as possible and information is not lost even in a deep structure.

The channel attention unit 220 is generated by applying a depth-separable convolution through a Global Average Pooling (GAP) block, a convolution block, a ReLU block, a convolution block, and a sigmoid block. Channel attention is applied to balance the output image quality and computational cost by compensating for loss of information.

The channel attention unit 220 focuses on required input feature points by utilizing interdependence between channels, compresses channel information using Global Average Pooling (GAP), and restores the input feature points through a convolution layer. In addition, in order to reinforce and restore input feature points for a part to be focused in the network, channel statistics are extracted through a gating mechanism.

의 경우 DCBlock은 다음과 같이 공식화할 수 있다: In DCBlock, input features and output features are supplied to each block undergoing residual learning. Input features

In this case, DCBlock can be formulated as follows:

는 깊이별 분리 가능 세션(Depthwise Separable Session)

, 채널 어텐션 세션(Channel Attention Session))

, 그리고 입력 특징인

으로 구성되어 있다. 이는 모두 잔차 학습(Residual Learning)으로 유기적인 연결이 되어있다. 입력된

과 이를

를 통해 출력된 특징인

과 함께 잔차 학습을 진행한다. 추가적으로

를 통해 추출된 특징을 채널 어텐션 세션에 공급해준

도 잔차 학습을 진행한다. 이에 해당되는 부분은 다음과 같이 공식화할 수 있다: DCBlock module means

Is a Depthwise Separable Session

, Channel Attention Session

, And the input feature

It consists of. These are all organically linked to residual learning. Entered

And this

The feature output through

And proceed with residual learning. Additionally

Provided the features extracted through the channel attention session

Also proceeds with residual learning. This can be formulated as follows:

은

과 곱연산을 진행하게 되고 게이팅(gaiting) 메커니즘으로 인해 특징에는 자연스럽게 채널의 통계치로 얻어진 집중점에 대해 입혀지게 되어 중요한 부분의 정보가 강화되고 정보 손실을 방지할 수 있다. 이 방법을 통 해 DCBlock은 이미지 간 변환 시 많은 수의 파라미터를 필요로 하는 학습모델의 파라미터 수를 줄일 수 있고, 기 존 이미지 간 변환을 학습하는 모델과 유사한 품질의 이미지를 생성할 수 있게 된다. Feature points extracted from channel attention

silver

The multiplication operation is performed, and due to the gating mechanism, the feature is naturally applied to the point of focus obtained as the statistics of the channel, thereby reinforcing the information of an important part and preventing information loss. Through this method, DCBlock can reduce the number of parameters of a learning model that requires a large number of parameters when converting between images, and it is possible to create an image of similar quality to a model that learns conversion between images.

3 is a flowchart illustrating an image conversion method of an efficient hostile-generated neural network using convolution and channel attention separable for each depth according to an embodiment of the present invention.

The proposed image conversion method for an efficient hostile generation neural network using separable convolution by depth and channel attention is information generated by applying depth separable convolution (310) and depth separable convolution to reduce the number of parameters. And applying (320) channel attention to a balance between the quality of the output image and the computational cost by compensating for the loss.

In step 310, depth-separable convolution is applied to reduce the number of parameters.

Information loss occurs when the number of parameters is reduced by applying depth-separable convolution. This loss of information can be overcome with residual linkage, but the result of the model is not as good as the image quality of the existing model because it is not alleviated in the conversion between images. Information loss occurs because deeply separable convolution is performed for each channel, and loss of features appearing in the entire portion is inevitable. Therefore, it was necessary to add a technique to compensate for information loss in order to achieve an optimal balance between the quality of the output image and the computational cost. Therefore, channel attention is applied so that necessary information is kept as much as possible and information is not lost even in a deep structure.

In step 320, channel attention is applied to balance the quality of the output image and computational cost by compensating for information loss caused by applying a depth-separable convolution. It will be described in more detail below with reference to FIGS. 4 and 5.

4 is a diagram for describing a convolution configuration that can be separated by depth according to an embodiment of the present invention.

As described above, in the present invention, in order to reduce the number of parameters, a separable convolution by depth consisting of a depthwise convolution as shown in FIG. 4(a) and a pointwise convolution as shown in FIG. 4(b) Use lution.

In depth convolution, there is a filter on the number of channels to extract spatial features, so the number of input and output channels is the same. The depth convolution can be expressed as:

은 깊이 컨볼루션의 커널 크기를 나타내고, i, j, m은 폭, 높이, 입력 채널을 나타내고

는 특징 맵을 나타낸다. 포인트 컨볼루션은 1x1 컨볼루션이며 필터의 크기는 1x1로 고정된다. 깊이 컨볼루션과 달리, 포인트 컨벌루션은 공간적 특징을 다루지 않고 채널에서만 수행된다. 이것은 DCBlock에서 계산량을 크게 줄이는데 도움이 된다. 표 1은 두 변환의 파라미터의 수와 계산 비용의 차이를 보여준다.here

Represents the kernel size of the depth convolution, i, j, and m represent the width, height, and input channel.

Represents a feature map. The point convolution is 1x1 convolution and the size of the filter is fixed at 1x1. Unlike depth convolution, point convolution does not deal with spatial features and is performed only in channels. This helps to significantly reduce the amount of computation in DCBlock. Table 1 shows the difference between the number of parameters and computation cost of the two transformations.

<Table 1>

는 커널 크기,

은 입력 및 출력 채널 크기를 각각 나타내고

는 입력 높이, 너비를 각각 나타낸다. 표 1에 나타낸 바와 같이, 표준 컨볼루션은 계산 비용이

이고 깊이 분리가능한 컨볼루션은

이다. 차이를 보기 위해 두 비용을 나누면 비용이

감소한 것으로 나타난다. 이는 상대적으로 깊은 모델이 사용되는 적대적 생성 신경망에서 비례 적으로 증가하는 파라미터의 수를 줄이는 데 큰 효과가 있다.here

Is the kernel size,

Represents the input and output channel sizes respectively

Represents the input height and width, respectively. As shown in Table 1, the standard convolution has a computational cost

And the depth separable convolution is

to be. If you divide the two costs to see the difference,

Appears to have decreased. This has a great effect in reducing the number of proportionally increasing parameters in a hostile generated neural network where a relatively deep model is used.

5 is a diagram for explaining a channel attention mechanism according to an embodiment of the present invention.

The channel attention mechanism is one of the variations of attention that focuses on required feature points by utilizing interdependencies between channels.

을 구하는 식이다: As can be seen in FIG. 5, a Global Average Pooling (GAP) block, a fully connected layer (FC) (that is, convolution) block, a ReLU block, and a fully connected layer (FC) (that is, convolution). )) Through a block, a sigmoid block, a channel attention 520 is applied to the input feature 510, the channel information is compressed using Global Average Pooling (GAP) and a convolution layer The feature point is restored through The following equation is the input feature point through GAP

Is the equation for:

Subsequently, the channel statistics are extracted through a gating mechanism. This effect reinforces and restores the features of the part to be focused within the network. Therefore, in order to maintain as many feature points as possible even in the deep model and to solve information loss due to convolution that can be separated in depth, the restored input feature points are output 530 using channel attention.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (8)

Applying a depth separable convolution to reduce the number of parameters; And
Step of applying channel attention to balance the quality of the output image and computational cost by compensating for information loss that occurs by applying depth-separable convolution
Including,
The step of applying a depth separable convolution to reduce the number of parameters,
Convolution for each depth is applied using the kernel size, width, height, input channel and feature map of the convolution for each depth, and point convolution that is performed only in the channel without dealing with spatial features is applied,
In the depth-specific convolution, the input features and output features are supplied to each block for residual learning, and the input features

In the case of, the convolution by depth is represented by the following equation,

Means convolution by depth

Is a Depthwise Separable Session

, Channel Attention Session

, And input features

And all of the above parameters are organically connected to residual learning, and the input

and

The feature output through

And, additionally

To supply the features extracted through the channel attention session

To proceed with residual learning
Image conversion method. delete The method of claim 1,
The step of applying channel attention to balance between the quality of the output image and the computation cost by compensating for information loss caused by applying a depth-separable convolution,
It focuses on required input feature points by utilizing interdependence between channels, compresses channel information using Global Average Pooling (GAP), and restores input feature points through a convolution layer.
Image conversion method. The method of claim 3,
Channel statistics are extracted through a gating mechanism in order to reinforce and restore the input feature points for the part to be focused in the network.
Image conversion method. Depth-by-depth separation unit for applying a depth-separable convolution to reduce the number of parameters; And
Channel attention unit that applies channel attention to balance the output image quality and computation cost by compensating for information loss that occurs by applying depth-separable convolution
Including,
Separation by depth,
Convolution for each depth is applied using the kernel size, width, height, input channel and feature map of the convolution for each depth, and point convolution that is performed only in the channel without dealing with spatial features is applied,
In the depth-specific convolution, the input features and output features are supplied to each block for residual learning, and the input features

In the case of, the convolution by depth is represented by the following equation,

Means convolution by depth

Is a Depthwise Separable Session

, Channel Attention Session

, And input features

And all of the above parameters are organically connected to residual learning, and the input

and

The feature output through

And, additionally

To supply the features extracted through the channel attention session

To proceed with residual learning
Image conversion device. delete The method of claim 5,
The channel attention department,
It focuses on required input feature points by utilizing interdependence between channels, compresses channel information using Global Average Pooling (GAP), and restores input feature points through a convolution layer.
Image conversion device. The method of claim 7,
The channel attention department,
Channel statistics are extracted through a gating mechanism in order to reinforce and restore the input feature points for the part to be focused in the network.
Image conversion device.

Images