KR102507460B1 - Cartoon background automatic generation method and apparatus - Google Patents

Abstract

A method and device for automatically generating a cartoon background are disclosed. The method of automatically generating a cartoon background includes: (a) generating a mask image after selecting an unnecessary area as a mask using an original image; (b) generating a reconstructed image from which a mask region is removed from the original image by applying the original image and the mask image to a first neural network-based reconstruction model; and (c) generating a cartoon background image by applying the reconstructed image to the learned second neural network-based style transition model to transform the target style image into an author's cartoon style reflecting the contour and abstraction characteristics of the target style image.

Description

Cartoon background automatic generation method and apparatus {Cartoon background automatic generation method and apparatus}

The present invention relates to a method and apparatus for automatically generating a cartoon background using an adversarial generative neural network.

With the development of the Internet, cartoons that were only available offline can now be easily accessed on PCs or smartphones. Webtoon is a compound word of web and cartoon, which is a cartoon that is serialized and distributed through the Internet. Most writers are suffering from deteriorating mental and physical health due to the type and amount of work that writers have to do (story design, storyboard writing, sketching, pen touch, coloring, background work, etc.).

The task of creating cartoon background images tends to be closer to the realm of repetitive simple labor than the realm of creation that requires creativity. Conventionally, in order to create a background image, a method of applying a picture effect to an original photo or tracing the original photo using an editing program such as Photoshop or Clip Studio was often used.

However, these conventional programs have a problem in that the artist's drawing style or style cannot be applied.

An object of the present invention is to provide a method and apparatus for automatically generating a cartoon background using an adversarial generation neural network.

The present invention enables automatic generation of a cartoon background reflecting the artist's painting style from photos, etc., thereby reducing work time due to simple labor, as well as automatically generating a cartoon background using an adversarial generative neural network that enables artists to focus more on their creative work It is to provide a method and an apparatus therefor.

According to one aspect of the present invention, a method for automatically generating a cartoon background using an adversarial generation neural network is provided.

According to one embodiment of the present invention, (a) generating a mask image after selecting an unnecessary area as a mask using an original image; (b) generating a reconstructed image from which a mask region is removed from the original image by applying the original image and the mask image to a first neural network-based reconstruction model; and (c) generating a cartoon background image by applying the reconstructed image to the learned second neural network-based style transition model to transform the target style image into a cartoon style reflecting the outline and abstraction characteristics of the target style image. An automatic generation method may be provided.

The target style image is a cartoon image of a specific artist, and the second neural network-based style transition model reflects the outline intensity value and the abstraction intensity value set through the style transition intensity UI as hyperparameters to determine the outline intensity and The cartoon background image can be created by changing the abstraction strength.

The second neural network-based style transition model is learned using the following equation,

는 카툰 배경 이미지와 타겟 스타일 이미지간의 적대적 손실 함수를 나타내며,

는 윤곽선 강도값과 추상화 강도값을 나타내는 초매개변수이며,

는 카툰 배경 이미지와 타겟 스타일 이미지간 각각에서 추출된 윤곽선 정보를 기반으로 도출되는 윤곽선에 대한 적대적 손실 함수이며,

는 카툰 배경 이미지와 타겟 스타일 이미지 각각에서 추출된 추상화 표현 정보를 기반으로 도출되는 추상화 표현에 대한 적대적 손실 함수이며,

는 카툰 배경 이미지와 타겟 스타일 이미지간의 불변성에 대한 손실 함수이다. here,

Represents the adversarial loss function between the cartoon background image and the target style image,

is a hyperparameter representing the contour strength value and the abstraction strength value,

is an adversarial loss function for the contour derived based on the contour information extracted from each of the cartoon background image and the target style image,

is an adversarial loss function for the abstract expression derived based on the abstract expression information extracted from each of the cartoon background image and the target style image,

is the loss function for the invariance between the cartoon background image and the target style image.

The style transition intensity UI is provided as a two-dimensional graph UI having an outline intensity axis and an abstraction intensity axis, and the outline intensity value and the abstraction intensity value are changed according to a point selected on the two-dimensional graph UI, and the changed outline intensity value and A cartoon background image obtained by converting the reconstructed image into a cartoon style in consideration of an abstraction strength value may be displayed on an output screen in real time.

According to another aspect of the present invention, an apparatus for automatically generating a cartoon background using an adversarial generating neural network is provided.

According to one embodiment of the present invention, a mask generator for generating a mask image after selecting an unnecessary region as a mask using an original image; a restoration unit generating a restoration image in which a mask region is removed from the original image by applying the original image and the mask image to a first neural network-based restoration model; and a style transition unit for generating a cartoon image by applying the reconstructed image to a learned second neural network-based style transition model and transitioning the restored image into a cartoon style reflecting the outline and abstraction characteristics of the target style image. It can be.

By providing a method and apparatus for automatically generating a cartoon background using an adversarial generative neural network according to an embodiment of the present invention, it is possible to automatically create a cartoon background reflecting an artist's painting style, thereby reducing work time due to simple labor. It can also free writers to focus more on their creative work.

1 is a flowchart illustrating a method for automatically generating a cartoon background according to an embodiment of the present invention.
2 is a diagram illustrating an original image and a mask image according to an embodiment of the present invention;
3 is a diagram for explaining a reconstructed image according to an embodiment of the present invention;
4 is a diagram showing a detailed structure of a generation module of a first neural network-based reconstruction model according to an embodiment of the present invention;
5 is a diagram showing a detailed structure of a discrimination module of a first neural network-based reconstruction model according to an embodiment of the present invention.
6 is a diagram showing a detailed structure of a second neural network-based style transition module according to an embodiment of the present invention;
7 to 9 are diagrams for explaining a style transition strength UI according to an embodiment of the present invention.
10 to 14 illustrate interface output screens for a process of generating a cartoon image from an original image according to an embodiment of the present invention.
15 is a block diagram schematically showing the internal configuration of an automatic cartoon background generating device according to an embodiment of the present invention.
16 is a diagram illustrating a cartoon background image created using an original image according to an embodiment of the present invention.

Singular expressions used herein include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some of the steps It should be construed that it may not be included, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a method for automatically generating a cartoon background according to an embodiment of the present invention, FIG. 2 is a diagram for explaining an original image and a mask image according to an embodiment of the present invention, and FIG. This is a diagram for explaining a reconstructed image according to an embodiment of the present invention, FIG. 4 is a diagram showing a detailed structure of a generation module of a first neural network-based restoration model according to an embodiment of the present invention, and FIG. This is a diagram showing the detailed structure of a discrimination module of a first neural network-based reconstruction model according to an embodiment of the present invention, and FIG. 6 shows a detailed structure of a second neural network-based style transition module according to an embodiment of the present invention. FIGS. 7 to 9 are diagrams for explaining a style transition intensity UI according to an embodiment of the present invention, and FIGS. 10 to 14 show a cartoon image from an original image according to an embodiment of the present invention. It is a diagram illustrating an interface output screen for the creation process.

In step 110, the apparatus 100 for automatically generating a cartoon background receives an original image and designates an unnecessary area in the original image to generate a mask image.

Here, the original image may be a photograph. In order to make such a photo image into a cartoon background, it is necessary to go through the process of changing it to suit the artist's drawing style or style after removing unnecessary areas taken with the background.

To this end, the device for automatically creating a cartoon background generates a mask image by designating an unnecessary area (ie, an object area such as a person) in the original picture after receiving an original picture to be changed into a cartoon background. The mask image includes only information on a designated unnecessary area in the original image, but may be used to change pixel values of the designated unnecessary area to 0.

Also, the apparatus for automatically generating a cartoon background may generate an input image in which an unnecessary region of the original image is covered with a mask. In (a) of FIG. 2, an original image is illustrated, in (b) of FIG. 2, a mask image is illustrated, and in (c) of FIG. 2, an input image is illustrated.

In step 115, the apparatus 100 for automatically generating a cartoon background generates a reconstructed image from which the mask image is removed by applying the input image and the mask image to a first neural network-based reconstruction model.

3(a) illustrates an original image, FIG. 3(b) illustrates an input image in which unnecessary areas are designated in the original image, and FIG. 3(c) shows the original image by removing unnecessary areas. A reconstructed image reconstructed with neighboring pixels is exemplified.

Hereinafter, a first neural network-based reconstruction model for generating a reconstructed image reconstructed with neighboring pixels by removing unnecessary regions from an original image will be described in detail.

The restoration model may be a Generative Adversarial Network (GAN) based model.

This reconstruction model includes a generating module and a discriminating module.

The generation module is composed of a two-level network having a first generation module and a second generation module, and may have an encoder-decoder structure.

The detailed structure of the generating module is as shown in FIG. 4 .

The first generation module receives an input image obtained by applying a mask image to an original image and a mask image, roughly restores the image, and outputs a first stage restored image.

As shown in FIG. 4, the first generation module has an encoder-decoder structure and is based on gated convolution and dilated gated convolution.

The second generation module may output the second stage reconstructed image by performing precise restoration using the surrounding area using the first stage reconstructed image, which is an output result of the first generation module. To this end, the second generation module has a contextual attention module, and through this, it is possible to generate a second stage reconstructed image by performing precise restoration using the surrounding area.

The generated image finally output through the generation module having a two-step network structure is determined whether it is real data or fake data through the discrimination module. The detailed structure of the determination module is as shown in FIG. 5 .

The discrimination module has a PatchGAN structure, and by discriminating real from fake in units of patches of a specific size rather than the entire area of the generated image, the learning speed is faster than the existing method of discriminating the entire image, and it has the advantage of obtaining high-resolution images. .

In addition, the discrimination module according to an embodiment of the present invention has an advantage of further stabilizing GAN learning by using spectral normalization.

The restoration module is learned through a loss function, Equation 1 is a loss function used to learn the generation module, and Equation 2 is a loss function used to learn the discrimination module.

는 초매개변수(hyper-parameter)를 나타내며, 사용자가 설정할 수 있다. 예를 들어,

와 같이 초매개변수가 설정될 수 있다. 이는 일 예일 뿐이며, 매개변수의 값은 적용 방법에 따라 다양하게 설정될 수 있다. here,

represents a hyper-parameter, which can be set by the user. for example,

Hyperparameters can be set as This is just an example, and the value of the parameter may be set in various ways according to the application method.

는 수학식 3과 같이 나타낼 수 있다.

can be expressed as in Equation 3.

는 복원 모듈을 통해 생성된 이미지(

)에서 마스크되지 않은 영역을 원본 이미지(

)로 바꾼 이미지이다. here,

is the image created through the restore module (

) in the original image (

) is the image changed to .

는 L1-손실 함수(L1 loss function)가 적용되며 원본 이미지(

)와 복원 모듈을 통해 생성된 이미지(

)이 일치되도록 유도하며, 수학식 4와 같다.

is applied to the L1-loss function and the original image (

) and the image created through the restore module (

) is induced to match, and is shown in Equation 4.

는 제1 단계 생성 이미지를 나타내고,

는 제2 단계 생성 이미지를 나타낸다. here,

Represents the first stage generated image,

denotes a second stage generated image.

는 수학식 5와 같이 도출될 수 있다. also,

Can be derived as in Equation 5.

는 L1 손실 함수를 적용하지만, ImageNet으로 사전 학습된 VGG-16 네트워크를 이용해 이미지를 특징 공간으로 투영한 후 계산된다.

는 p번째 레이어의 특징맵을 나타내고, pool1, pool2, pool3 레이어가 사용될 수 있다. 또한,

는

의 원소의 개수를 나타낸다. here,

applies the L1 loss function, but is computed after projecting the image into feature space using the VGG-16 network pre-trained with ImageNet.

denotes a feature map of the p-th layer, and pool1, pool2, and pool3 layers may be used. also,

Is

represents the number of elements in

는 수학식 6을 이용하여 도출될 수 있다.

Can be derived using Equation 6.

는 두 이미지 사이의 스타일 차이를 줄여주는 함수이다. 스타일이란 서로 다른 특징맵간의 상관관계로 정의하기로 한다. 사전 학습된 VGG-16 네트워크를 이용해 서로 다른 이미지의 특징맵을 추출하고 특징맵 간의 상관관계를 계산하기 위해 그람 행렬(gram matrix)이 사용될 수 있다. here,

is a function that reduces the style difference between two images. The style will be defined as a correlation between different feature maps. A gram matrix may be used to extract feature maps of different images using a pretrained VGG-16 network and calculate a correlation between feature maps.

는 그람 행렬이다.

is the Gram matrix.

The image restored through the restoration module is shown in (c) of FIG. 3 . As shown in FIG. 3, a restored image suitable for use as a cartoon background can be created by specifying an unnecessary area in the original image to set a mask, restoring the mask portion using pixels around the original image, and removing the unnecessary area. there is.

The restored image is a restored photographic image, and is not suitable for use as a cartoon background as it does not reflect the artist's painting style or style.

Accordingly, in step 120, the apparatus 100 for automatically generating a cartoon background applies the reconstructed image to a second neural network-based style transition model to reflect the outline and abstract expression of the target style image to generate a cartoon image transitioned into a cartoon style.

The second neural network-based style transition model is a model generated to transfer a reconstructed image into a cartoon style based on the outline and abstract expression of the target style image. Hereinafter, the second neural network-based style transition model will be described in more detail with reference to FIG. 6 .

The second neural network-based style transfer model is a GAN-based model. Accordingly, the second neural network-based style transition model includes a generating module and a discriminating module, as shown in FIG. 6 .

The generating module may generate a cartoon style image by receiving the reconstructed image reconstructed through the first neural network-based restoration model and reflecting the outline and abstract expression of the target style image. To this end, the generation module has a U-Net structure and can be composed of a convolution layer, a leaky ReLu, and a bilinear-resize layer.

The discrimination module is a means for making the styles of the two images similar by discriminating between the cartoon image generated by the generation module and the target style image. The discrimination module has a PatchGAN structure, and GAN learning can be further stabilized by using spectral normalization.

The second neural network-based style transfer model can reflect the painting style of each artist by clustering target style images for each artist and learning them.

The second neural network-based style transition model according to an embodiment of the present invention is configured to learn outlines and abstract expressions, which are typical characteristics of cartoons.

Unlike general images or photos, cartoons have clear pen lines, and objects are identified through them. Accordingly, the second neural network-based style transfer model may be trained to express the pen lines of the artist's cartoon after extracting outline information from the target style image. For example, contour information may be extracted through a Canny Edge detector. We will briefly explain this.

First, the apparatus 100 for automatically generating a cartoon background reduces noise in an image using a Gaussian filter, and then differentiates the horizontal and vertical directions of the image to find an edge. Here, an edge may be defined as a part where the intensity of an image rapidly changes. In addition, since the device for automatically generating a cartoon background may have a region that is detected even though it is not an edge, it is possible to remove a weak edge that is not the maximum value by comparing and determining the edge strength of the current pixel with the edge strength in the positive and negative directions. . In addition, the apparatus for automatically generating a cartoon background may extract only the outline from the image by performing a process of finding a certain edge through setting a threshold.

In an embodiment of the present invention, the second neural network-based style transition model assumes that the Canny Edge detector is used, and the explanation is centered on this, but this is for convenience of understanding and explanation, and the contour feature does not necessarily use only the Canny Edge detector. Any known edge detector can be applied.

Another characteristic of cartoons is that objects are expressed in abstract rather than substance. The detailed texture of an object disappears and the surface is expressed smoothly. Accordingly, the second neural network-based style transfer model may be pretrained by extracting abstract features of the target style image.

Abstract expression features may be extracted using, for example, a deep guided filter. The deep guide filter can prevent image distortion due to loss of information around the contour while providing a smoothing effect to the image.

As described above, the second neural network-based style transfer model may learn a loss function in consideration of contour lines and abstraction characteristics. The loss function of the second neural network-based style transfer model is shown in Equation 7.

는 카툰 이미지와 타겟 스타일 이미지간의 적대적 손실 함수를 나타낸다. here,

represents the adversarial loss function between the cartoon image and the target style image.

는 수학식 8과 같이 도출될 수 있다.

Can be derived as in Equation 8.

는 타겟 스타일 이미지(즉, 작가의 카툰 이미지)를 나타내며,

는 제2 신경망 기반 스타일 전이 모델을 통해 생성된 카툰 이미지를 나타낸다. 즉, 제2 신경망 기반 스타일 전이 모델은 입력되는 복원 이미지를 이용하여

와 같은 이미지를 생성하여 판별 모듈을 속이려 하고, 판별 모듈은

와

를 판별할 수 잇다. here,

represents the target style image (ie, the author's cartoon image),

represents a cartoon image generated through the second neural network-based style transfer model. That is, the second neural network-based style transition model uses the input restored image

Trying to trick the discrimination module by generating an image such as , and the discrimination module

and

can determine

이

와 비슷해지도록 학습될 수 있다. The second neural network-based style transfer model is a GAN-based model and is a deep learning model in which the generation module and the discrimination module learn while fighting against each other.

this

can be learned to be similar to

는 초매개변수로, 윤곽선 강도값과 추상화 강도값을 나타낸다.

을 조절함으로써 도 7 내지 도 9에 도시된 바와 같이 스타일 조절이 가능하도록 할 수 있다.

is a hyperparameter, representing the contour intensity value and the abstraction intensity value.

It is possible to adjust the style as shown in FIGS. 7 to 9 by adjusting the .

는 윤곽선 강도를 조절하는 초매개변수로 값이 커지면 윤곽선이 진해지고 값이 작아지면 윤곽선이 연해진다.

는 추상화 강도를 조절하는 초매개변수로 값이 커지면 구조물 형태가 추상적이고 텍스처가 줄며 값이 작아지면 텍스처가 살아나게 된다.

is a hyperparameter that controls the strength of the outline. As the value increases, the outline becomes thicker, and as the value decreases, the outline becomes softer.

is a hyperparameter that controls the intensity of abstraction. As the value increases, the structure shape becomes abstract and the texture decreases, and as the value decreases, the texture becomes alive.

In this way, in changing the reconstructed image into a cartoon style by reflecting the outline and abstraction characteristics of the target style image, there is an advantage in that the degree of freedom in editing can be increased by adjusting the outline intensity value and the abstraction intensity value.

는 카툰 이미지와 타겟 스타일 이미지의 윤곽선에 대한 적대적 손실 함수를 나타낸다.

denotes the adversarial loss function for the contours of the cartoon image and the target style image.

는 수학식 9와 같이 나타낼 수 있다.

Can be expressed as in Equation 9.

는 이미지에서 추출된 윤곽선 정보(특징)을 나타낸다. 즉,

는 카툰 이미지(

)에서 추출한 윤곽선 특징이 타겟 스타일 이미지(

)에서 추출한 윤곽선 특징과 비슷해지도록 적대적으로 학습하는 손실 함수이다. here,

represents the contour information (feature) extracted from the image. in other words,

is a cartoon image (

), the contour features extracted from the target style image (

) is a loss function that learns adversarially to resemble the contour features extracted from

는 카툰 이미지와 타겟 스타일 이미지의 추상화 표현에 대한 적대적 손실 함수를 나타낸다.

denotes the adversarial loss function for abstract representations of cartoon images and target-style images.

는 수학식 10과 같이 나타낼 수 있다.

Can be expressed as in Equation 10.

는 딥 가이드 필터를 이용하여 추출된 추상화 특징을 나타낸다.

는 카툰 이미지(

)에서 추출한 추상화 특징이 타겟 스타일 이미지(

)에서 추출한 추상화 특징과 비슷해지도록 적대적으로 학습하는 손실 함수이다. here,

represents an abstraction feature extracted using a deep guide filter.

is a cartoon image (

), the abstraction features extracted from the target style image (

) is a loss function that learns adversarially to resemble the abstraction features extracted from

는 수학식 11과 같이 나타낼 수 있다.

Can be expressed as in Equation 11.

는 타겟 스타일 이미지로 전이된 이미지와 복원된 카툰 이미지가 의미상 불변성을 갖도록 사용된다.

is used so that the image converted to the target style image and the restored cartoon image have semantic invariance.

As described above, the second neural network-based style transition model may create a cartoon background (image) by transitioning the reconstructed image into a cartoon style by reflecting the outline and abstraction characteristics of the target style image.

10 to 14 are diagrams illustrating interface output screens for a process of generating a cartoon image from an original image.

As shown in FIG. 10 , an original image to be converted into a cartoon image among a plurality of images may be loaded and displayed on one area of the output screen by selecting Load original image.

11 shows an interface for creating a mask image from an original image. When the mask creation button is pressed, a tool tool that can be selected by the user is output, and an unnecessary area can be specified by selecting any one tool from the tool tool. .

When the mask creation button is pressed again in a state in which the unnecessary area designation is completed, a mask image including only the unnecessary area is created as shown in FIG.

When the image restoration button is pressed as shown in FIG. 13, the input image and the mask image are applied to the first neural network-based restoration model to remove the mask, and then the resulting image in which the mask area is naturally restored based on the surrounding pixels of the input image (restoration image) may be output through one area of the output screen.

As illustrated in FIG. 7, when the style transition button is pressed, a bar for selecting a cartoon style image for each artist is displayed, and then when a target style image is selected, a style transition strength UI for setting style strength is provided in the form of a two-dimensional graph. do. Here, one axis of the two-dimensional graph represents the intensity of the outline, and the other axis represents the intensity of abstraction. A contour strength value and an abstraction strength value may be determined according to points selected from the 2D graph.

In this way, when the contour strength value and the abstraction strength value are determined, the reconstructed image is applied to the second neural network-based style transition model, and the cartoon style is reflected by the contour line and abstraction characteristics of the target style image through the second neural network-based style transition model. A cartoon image transitioned to may be generated and displayed on one area of the output screen.

You can change the outline and abstract expression of a cartoon image by adjusting the outline intensity value and abstraction intensity value in various ways through the style transition intensity UI.

That is, FIG. 7 shows an example in which a cartoon image generated accordingly is displayed on one area of an output screen when medium-level outline strength values and abstraction strength values are selected. FIG. 8 shows an example in which outline strength values are lowered and abstraction strength An example in which a cartoon image generated by increasing the value is displayed on one area of the output screen is shown. In addition, FIG. 9 shows an example in which a cartoon image generated when an outline strength value is increased and an abstraction strength value is decreased is displayed on one area of an output screen.

That is, based on the second neural network-based style transition model according to an embodiment of the present invention, there is an advantage in that the outline strength value and the abstraction strength value can be finely adjusted so as to create a cartoon background in a style desired by the artist.

In order to save the cartoon image converted into a desired style, as shown in FIG. 14, the cartoon background image can be finally saved by pressing the image save button on the interface.

15 is a block diagram schematically showing the internal configuration of an apparatus for automatically generating a cartoon background according to an embodiment of the present invention, and FIG. 16 is a cartoon background image generated using an original image according to an embodiment of the present invention. It is an illustrated drawing.

Referring to FIG. 15, the apparatus 100 for automatically generating a cartoon background according to an embodiment of the present invention includes a mask generating unit 1510, a restoration unit 1515, a style transition unit 1520, a memory 1525, and a processor ( 1530).

The mask generator 1510 is a means for generating a mask image including only the mask area after designating an unnecessary object area in the original image as a mask area.

The restoration unit 1515 includes a first neural network-based restoration model, applies the original image and the mask image to the first neural network-based restoration model, removes the mask region from the original image, and then restores the original image using neighboring pixels. It is a means for generating a restoration image.

Since the detailed structure and functions of the first neural network-based restoration model are the same as those described above, duplicate descriptions will be omitted.

The style transition unit 1520 is a means for generating a cartoon background image by transitioning the restored image into the artist's cartoon style. The style transition unit 1520 includes a second neural network-based style transition model. The style transition unit 1520 applies the reconstructed image to the learned second neural network-based style transfer model and transforms it into a cartoon style that reflects the outline and abstraction characteristics of the author's cartoon style image (ie, the target style image), thereby converting the cartoon background image can create At this time, the second neural network-based style transfer model receives the outline intensity value and the abstraction intensity value as hyperparameters and further reflects them to generate a cartoon background image in which the cartoon style image outline and abstraction characteristics of the artist are reflected. can do.

Since the second neural network-based style transition model is the same as described above, duplicate descriptions will be omitted.

The memory 1525 is a means for storing instructions for performing the method of automatically generating a cartoon background according to an embodiment of the present invention.

The processor 1530 includes internal components (eg, a mask generator 1510, a restoration unit 1515, and a style transition unit 1520) of the automatic cartoon background generation device 100 according to an embodiment of the present invention. , memory 1525, etc.).

According to an embodiment of the present invention, the apparatus for automatically generating a cartoon background includes a first neural network-based restoration model and a second neural network-based style transition model. Here, the first neural network-based restoration model and the second neural network-based style transition model are based on a deep learning model, and such a deep learning model has a large number of parameters and a large amount of computation, so it takes a long time to calculate and takes up a lot of storage space. .

Moreover, since it is composed of the first neural network-based reconstruction model and the second neural network-based style transition model, it takes a long time to calculate and occupies a large amount of storage space. Accordingly, according to another embodiment of the present invention, it can be applied in a notebook or mobile environment by reducing the weight.

Methods such as model distillation, channel pruning, and quantization can be used for model compression.

Model distillation is a method of transferring the knowledge of a large network (teacher network) that has been well trained in advance to a small network (student network) that actually wants to use it. The large network uses a first neural network-based restoration model and a second neural network-based style transfer model. It is a learned model that includes, and the small network can be a model to be applied to a cartoon background generating program. In other words, model distillation can make the two networks similar by including the difference between the classification result of the large network and the small network in the loss function.

Since the number of channels in a deep learning model is directly related to the number of parameters and the amount of computation, a considerable amount of resources can be wasted. Therefore, channel pruning can also lighten the model by removing channels that have little effect on the results.

Quantization is a technique of converting input values expressed in precise and detailed units into values in simpler units. It is widely used in artificial intelligence fields such as deep learning techniques to improve power efficiency while reducing the amount of computation. For example, the continuous color spectrum of the real world can be expressed in discontinuous colors, black and white images can be expressed with 1 bit per pixel, and color images can be expressed with 24 bits per pixel to reduce weight.

The apparatus 100 for automatically generating a cartoon background as described above can speed up computation by applying a lightweight model, use a smaller storage space, and can be applied to a laptop or mobile environment without a GPU.

16 is a cartoon style by the apparatus 100 for automatically generating a cartoon background according to an embodiment of the present invention by applying the reconstructed image to a second neural network-based style transition model to reflect the outline characteristics and abstraction characteristics of the artist's cartoon style image. A cartoon background image generated by transition is exemplified.

As described above, after removing unnecessary object areas from a photo, etc., the restored image is transformed into a cartoon style that reflects the artist's cartoon style (outline features and abstraction features) to automatically generate a cartoon background image, thereby reducing the artist's work time. There are advantages that can be

Devices and methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in computer readable media. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - Includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media and ROM, RAM, flash memory, etc. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been looked at mainly by its embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

100: automatic cartoon background generating device
1510: mask generating unit
1515: Restoration unit
1520: style transition unit
1525: memory
1530: processor

Claims (8)

(a) generating a mask image after selecting an unnecessary area as a mask using an original image;
(b) generating a reconstructed image from which a mask region is removed from the original image by applying the original image and the mask image to a first neural network-based reconstruction model; and
(c) generating a cartoon background image by applying the reconstructed image to the learned second neural network-based style transition model and transitioning it into an author's cartoon style reflecting the outline and abstraction characteristics of the target style image;
The second neural network-based style transition model generates the cartoon background image by changing the outline strength and abstraction strength of the target style image by reflecting the outline strength value and the abstraction strength value set through the style transition strength UI as hyperparameters. but
The style transition intensity UI is provided as a two-dimensional graph UI having an outline intensity axis and an abstraction intensity axis,
Depending on the point selected on the 2D graph UI, the contour intensity value and the abstraction intensity value are changed.
The method of automatically generating a cartoon background according to claim 1 , wherein a cartoon background image converted into a cartoon style of the reconstructed image is displayed on an output screen in real time in consideration of the changed outline strength value and abstraction strength value.
According to claim 1,
The method of automatically generating a cartoon background, characterized in that the target style image is a cartoon image of a specific artist.
delete According to claim 1,
The method of automatically generating a cartoon background, characterized in that the second neural network-based style transition model is learned using the following equation.

here,

Represents the adversarial loss function between the cartoon background image and the target style image,

is a hyperparameter representing the contour strength value and the abstraction strength value,

is an adversarial loss function for the contour derived based on the contour information extracted from each of the cartoon background image and the target style image,

is an adversarial loss function for the abstract expression derived based on the abstract expression information extracted from each of the cartoon background image and the target style image,

Is a loss function for invariance between a cartoon background image and a target style image.
delete A computer-readable recording medium on which program code for performing the method of any one of claims 1, 2, or 4 is recorded.
a mask generator for generating a mask image after selecting an unnecessary area as a mask using an original image;
a restoration unit generating a restoration image in which a mask region is removed from the original image by applying the original image and the mask image to a first neural network-based restoration model; and
A style transition unit for generating a cartoon image by applying the reconstructed image to a learned second neural network-based style transition model and transitioning the restored image into a cartoon style reflecting the outline characteristics and abstraction characteristics of the target style image,
The second neural network-based style transition model reflects the outline strength value and the abstraction strength value set through the style transition strength UI as hyperparameters to change the outline strength and abstraction strength of the target style image to generate a cartoon background image, ,
The style transition intensity UI is provided as a two-dimensional graph UI having an outline intensity axis and an abstraction intensity axis,
Depending on the point selected on the 2D graph UI, the contour intensity value and the abstraction intensity value are changed.
The apparatus for automatically generating a cartoon background according to claim 1 , wherein a cartoon background image obtained by converting the reconstructed image into a cartoon style in consideration of the changed outline strength value and abstraction strength value is displayed on an output screen in real time.
delete

Images