KR102197653B1 - Method, system and computer program for sketch-to-line translation - Google Patents

Abstract

According to one embodiment of the present invention, provided is a sketch-line conversion system based on style capable of converting a rough sketch image into a visually stunning line art image. The sketch-line conversion system based on style comprises: a pair data generation unit synthesizing a sketch image from a given line image to generate sketch-line combination pair data; a destyling unit extracting a line structure by removing a texture from a rough sketch image using a destyled model trained by the pair data; and a styling unit adding a style by applying a line style to the image having the extracted line structure using a styled model.

Description

Sketch-to-line transformation method, system and program {METHOD, SYSTEM AND COMPUTER PROGRAM FOR SKETCH-TO-LINE TRANSLATION}

The present invention relates to a sketch-line conversion method, an apparatus, and a program, and more particularly, a sketch-line conversion method, a system and a system capable of removing noise while preserving the details of a sketch using a destylization and stylization model, and It's about the program.

Most artist paintings are completed after a sketch-line translation process, which draws only the desired line from most sketches, is almost essential. However, since the sketch-line conversion work is close to a simple work that is additionally given after the sketch work, which is a creative activity, a technique to automatically perform the sketch-line conversion work is being studied. Existing sketch-line conversion studies automatically generated line images from rough sketches using neural networks, but there were problems of missing important details or preserving noise.

On the other hand, in order to derive a useful model through machine learning, it is necessary to give a computer a sufficient amount of data and a learning process to find common patterns or functions there. In particular, unlike unsupervised learning, supervised learning infers patterns or functions from training data, and securing rich training data is an essential factor in generating accurate models. .

An object of the present invention is to extract only a line structure from a sketch image using a destylization model and apply a line style to the line structure using a stylization model to perform sketch-line conversion.

Another object of the present invention is to synthesize sketch-line pair data and use it as training data in order to train a sketch-line transformation model.

According to an embodiment of the present invention, there is provided a pair data generator for synthesizing a sketch image from a given line image to generate sketch-line combination pair data; A destylization unit for extracting a line structure by removing texture from a rough sketch image using the destylization model trained by the paired data; A styling unit for adding a style by applying a line style to the extracted line structure image using a styling model; A sketch-line conversion system based on a style including a is provided.

In the present invention, the pair data generator generates a vector image from a raster line image, defines a curve using a plurality of control points, generates a new curve by adding noise to the plurality of control points, and generates a plurality of the curves. By dividing it into two small curves and adding noise, you can create a noise stroke.

In the present invention, the destylization unit may include a texture removal unit to remove a texture and a line structure extraction unit to extract a line structure.

In the present invention, the destylization unit may further include a line normalization unit for normalizing the line width so that the line width is not deformed due to data augmentation of the ground truth line image.

In the present invention, the texture removal unit may remove a background color or a sketch line color from the sketch.

In the present invention, the texture remover may be trained using texture augmented composite pair data, and the line structure extractor may be trained using composite pair data.

In the present invention, the line structure may be a center of a line irrelevant to the style of the rough sketch.

According to another embodiment of the present invention, a destylization step of extracting a line structure by removing a texture from a rough sketch image using a destylization model; And a styling step of adding a style by applying a line style to the image having the extracted line structure using a styling model. A sketch-line conversion method based on a style including a is provided.

In the present invention, the destyling step comprises: a texture removing step of removing a texture; And a line structure extraction step of extracting the line structure.

In the present invention, the destyling step further includes a line normalization step of normalizing the line width so that the line width is not deformed due to data augmentation of the ground truth line image. I can.

In the present invention, the texture removal step is trained using texture augmented composite pair data, and the line structure extraction step is trained using composite pair data.

In the present invention, the texture removing step may remove a background color or a sketch line color from the sketch.

According to another embodiment of the present invention, a pair data generation step of generating sketch-line combination pair data by synthesizing a sketch image from a given line image; A destylization step of extracting a line structure by removing texture from a rough sketch image using a destylization model trained by the paired data; And a styling step of adding a style by applying a line style to the image having the extracted line structure using a styling model. A sketch-line conversion method based on a style including a is provided.

In the present invention, in the step of generating pair data, a vector image is generated from a raster line image, a curve is defined using a plurality of control points, noise is added to the plurality of control points to generate a new curve, and the curve is Noise strokes can be generated by dividing into a plurality of small curves and adding noise.

In the present invention, the destyling step comprises: a texture removing step of removing a texture; And a line structure extraction step of extracting the line structure.

In the present invention, the destyling step further includes a line normalization step of normalizing the line width so that the line width is not deformed due to data augmentation of the ground truth line image. I can.

A computer program recorded on a computer-readable recording medium is further provided for carrying out the method according to the present invention.

According to the present invention, by performing the destyling step and the styling step, a rough sketch image can be converted into a visually excellent line art image.

In addition, according to the present invention

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating an internal configuration of a user terminal and a server according to an embodiment of the present invention.
3 is a diagram illustrating an internal configuration of a processor according to an embodiment of the present invention.
4 is a diagram illustrating a sketch-line conversion method according to an embodiment of the present invention in time series.
5 is a diagram illustrating a process of creating a sketch from a clean line image according to the method of creating a synthetic sketch of the present invention.
6 is for explaining a destylization and stylization model according to an embodiment of the present invention.
7 illustrates the effects of each step of the destylization model.
8 is an example for explaining the effect of the styling model of the present invention.

DETAILED DESCRIPTION OF THE INVENTION The detailed description of the present invention to be described later refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable those skilled in the art to practice the present invention. It is to be understood that the various embodiments of the present invention are different from each other, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be changed from one embodiment to another and implemented without departing from the spirit and scope of the present invention. In addition, it should be understood that the positions or arrangements of individual elements in each embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the detailed description to be described below is not made in a limiting sense, and the scope of the present invention should be taken as encompassing the scope claimed by the claims of the claims and all scopes equivalent thereto. Like reference numerals in the drawings indicate the same or similar elements over several aspects.

1 is a diagram illustrating an example of a network environment of a sketch-line conversion system according to an embodiment of the present invention.

The network environment of FIG. 1 shows an example including a plurality of user terminals 110, 120, 130, 140, a server 150, and a network 160. 1 is an example for explaining the invention, and the number of user terminals or the number of servers is not limited as shown in FIG. 1.

The plurality of user terminals 110, 120, 130, and 140 may be a fixed terminal implemented as a computer device or a mobile terminal. Examples of the plurality of user terminals 110, 120, 130, and 140 include smart phones, mobile phones, navigation, computers, notebook computers, digital broadcasting terminals, personal digital assistants (PDAs), and portable multimedia players (PMPs). ), tablet PC, etc. For example, the user terminal 1 110 may communicate with other user terminals 120, 130, 140 and/or the server 150 through the network 160 using a wireless or wired communication method.

The communication method is not limited, and not only a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 160 may include, but also a short-range wireless communication between devices is included. I can. For example, the network 160 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , Internet, and the like. In addition, the network 160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. Not limited.

The server 150 is a computer device or a plurality of computer devices that communicates with a plurality of user terminals 110, 120, 130, 140 and a network 160 to provide commands, codes, files, contents, services, etc. Can be implemented. The server 150 of the present invention may perform sketch-line conversion at a request from a user terminal.

For example, the server 150 may obtain a rough sketch image from the user terminal 1 110 accessed through the network 160 to generate a line art image. Under the control of an operating system (OS) included in the user terminal 1 (110) and at least one program (for example, a browser or an installed application), a rough sketch image is generated by the server 150 by accessing the server 150 Or, you can request a line art image, or you can ask to train a sketch-to-line transformation model. As another example, the server 150 may establish a communication session for data transmission/reception, and may route data transmission/reception between a plurality of user terminals 110, 120, 130, and 140 through the established communication session.

On the other hand, in the sketch-line conversion system of the present invention, the sketch is a pre- or intermediate stage of the work of creating a product design or line art such as a cartoon. Therefore, sketches are a set of lines drawn by the artist themselves and are often in a state prior to completion, and sketches play an important role in the early stages of many design works such as cartoons, movies and product design. After the sketch design is completed, the artist needs a sketch-to-line conversion task that simplifies the sketch into clean lines due to the roughness and overlap of the lines included in the sketch. Since the sketch-line conversion work is often unnecessary time added after initial creation, models for performing this sketch-line conversion work by machine learning have been developed recently. This sketch-line conversion model has a difficult point because it has to distinguish between the parts to be preserved and the parts that are not, and there is a difficulty in distinguishing the detail structure and noise lines.

In more detail, the existing sketch-line conversion methods are based on image structure conversion or style transfer, and since it is not easy to distinguish detailed structural details or noisy lines, the sketch is converted into a high-quality line image. There was a difficulty in doing it.

According to an embodiment of the present invention, a new style-based sketch-line conversion method may be provided using a two-step approach including a destylization model and a stylization model. In more detail, according to an embodiment of the present invention, in the destylization step, a sketch structure is simplified while removing a texture by using pair data (pair data consisting of sketch-line data) while maintaining content. . In the present invention, to overcome the shortage of pair data, a method of generating sketch-line data based on curve deformation is proposed. In addition, according to an embodiment of the present invention, in the styling step, styling may be performed based on an Augmented CycleGAN (Cycle Generative Adversarial Nets (GAN)) for the styled line expression. In addition, in the present specification, actual experimental results for the sketch-line conversion method of the present invention are introduced to show that the sketch-line conversion method of the present invention is superior in quantitative and qualitative measurement criteria.

In more detail, the present invention first extracts a content image representing a line structure from a sketch image, and then applies a style to a clean line image. In the previous studies, after encoding the content image into a feature map using a neural network, styling it using quadratic statistics, or indirectly learning a decoder function. However, referring to existing studies, even removing a small portion of content detail may lead to a large visual defect. That is, according to the existing technologies, some details may be lost due to a sketch-line conversion operation, and some noises to be disappeared may be rather preserved. That is, it is difficult to model which part of the image is a detail representation and which part is noise using only distribution matching using unsupervised learning.

Accordingly, according to the present invention, a model for transformation including a stylization model and a destylization model can be provided as a model for sketch-line style transformation. The destylized model may extract only the line structure by removing the sketch style from the sketch image, and the stylized model may obtain a line art image with style added by applying a line style to the line structure. In addition, the present invention processes the output of the destylization job as a content image rather than a final output image. Since the purpose of the sketch simplification is also to remove the rough line style from the sketch image, the present invention interprets this work as a destylization work in order to more clearly show the connection with the styling work. The destylization model of the present invention has the following differences when compared with existing studies.

First, the present invention uses automatically generated sketch-line pair data instead of manually generated data. Second, the present invention removes the texture before destyling the rough sketch, so that it may not focus on information irrelevant to the line structure.

In addition, in the styling step of the present invention, the content image extracted using the extended CycleGAN may be styled in various line styles. The present invention uses the destylized model and the stylized model in sequence, so that the present invention can convert rough sketches into visually outstanding line art images.

In summary, the present invention can provide a sketch-line conversion operation in consideration of the line style of the final output image. In addition, the present invention can provide a destylization model using texture removal and synthesis pair data. Further, the present invention can provide a model for synthesizing paired data for training a destylization model. That is, the present invention considers not only the line structure but also the style, which is an important factor that determines the mood and impression of a picture, as compared with the existing sketch simplification work. In addition, the existing work relies on manually generated sketch-pair data, but the present invention can use automatically generated pair data.

According to an embodiment of the present invention, a new sketch-line conversion method based on style is provided using a two-step approach including a model of destylization and stylization.

2 is a block diagram illustrating an internal configuration of a user terminal and a server according to an embodiment of the present invention.

In FIG. 2, the internal configuration of the user terminal 1 110 as an example of one user terminal and the server 150 as an example of one server is described. Other user terminals 120, 130, and 140 may also have the same or similar internal configuration.

The user terminal 1 110 and the server 150 may include memories 211 and 221, processors 212 and 222, communication modules 213 and 223, and input/output interfaces 214 and 224. The memories 211 and 221 are computer-readable recording media, and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. In addition, the memory 211 and 221 may store an operating system and at least one program code (for example, a browser installed and driven in the user terminal 1 110 or a code for the aforementioned application). These software components may be loaded from a computer-readable recording medium separate from the memories 211 and 221 using a drive mechanism. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memories 211 and 221 through communication modules 213 and 223 other than a computer-readable recording medium. For example, at least one program is a program installed by files provided through the network 160 by a file distribution system (for example, the server 150 described above) for distributing the installation files of developers or applications (example It may be loaded into the memories 211 and 221 based on the application implementing the sketch-line conversion method described above.

The processors 212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processors 212 and 222 by the memories 211 and 221 or the communication modules 213 and 223. For example, the processors 212 and 222 may be configured to execute a command received according to a program code stored in a recording device such as the memories 211 and 221. In particular, the processor 222 of the server 150 converts the rough sketch image obtained from the user terminal 110 into a line art image using a destylization and stylization model, and trains a destylization and stylization model. can do. A detailed configuration and operation of the processor 222 will be described later.

The communication modules 213 and 223 may provide a function for the user terminal 1 110 and the server 150 to communicate with each other through the network 160, and other user terminals (for example, user terminal 2 120) Alternatively, a function for communicating with another server (for example, the server 150) may be provided. As an example, a request generated by the processor 222 of the user terminal 1 110 according to a program code stored in a recording device such as the memory 211 is transmitted through the network 160 under the control of the communication module 213. 150). Conversely, control signals, commands, contents, files, etc. provided under the control of the processor 222 of the server 150 are transmitted via the communication module 223 and the network 160 to the communication module of the user terminal 1 110 ( 213) through the user terminal 1 (110). For example, a control signal or command of the server 150 received through the communication module 213 may be transmitted to the processor 222 or the memory 211, and the user terminal 1 110 may transmit contents or files. It may be stored in a storage medium that may further contain.

The input/output interfaces 214 and 224 may be means for interfacing with the input/output device 215. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display for displaying a communication session of an application. As another example, the input/output interface 214 may be a means for interfacing with a device in which input and output functions are integrated into one, such as a touch screen. As a more specific example, the processor 222 of the user terminal 1 110 is configured by using data provided by the server 150 or the user terminal 2 120 in processing the command of the computer program loaded in the memory 211 The service screen or content to be displayed may be displayed on the display through the input/output interface 214.

In addition, in other embodiments, the user terminal 1 110 and the server 150 may include more components than those of FIG. 2. However, there is no need to clearly show most of the prior art components. For example, the user terminal 1 110 may be implemented to include at least some of the input/output devices 215 described above, or other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, a database, etc. It may also contain more elements.

3 is a diagram illustrating an internal configuration of a processor according to an embodiment of the present invention.

The processor 222 may include an application that executes a sketch-line conversion model capable of obtaining a sketch image and returning a line art image. In addition, the processor 222 may train a sketch-line transformation model using synthetic data. The configuration for performing the sketch-line conversion function according to an embodiment of the present invention within the processor 222 is a pair data generation unit 310, a destylization unit 320, and a styling unit 330 as shown in FIG. 3. ) Can be included. Depending on the embodiment, components of the processor 222 may be selectively included or excluded from the processor 222. Further, according to an embodiment, components of the processor 222 may be separated or merged to express the function of the processor 222.

Here, the components of the processor 222 may be expressions of different functions of the processor 222 executed by the processor 222 according to an instruction provided by the program code stored in the server 100. . The processor 222 trains the destylization model and the stylization model with the synthesized pair data using the pair data generation unit 310, the destylization unit 320 and the stylization unit 330, and uses the trained model. You can perform the sketch-line conversion function.

The components of the processor 222 and the processor 222 may control the server 150 to perform the steps (S1 to S4) included in the sketch-line conversion method of FIG. 4 to be described later. For example, the processor 222 and the components of the processor 222 may be implemented to execute an instruction (instruction) according to the code of the operating system included in the memory 211 and the code of at least one program.

4 is a diagram illustrating a sketch-line conversion method according to an embodiment of the present invention in time series. Hereinafter, a sketch-line conversion method, system, and computer program of the present invention will be described in detail with reference to FIGS. 3 and 4.

As described above, the sketch-line conversion system and method of the present invention includes a destylization model for extracting a line structure from a sketch and a stylization model for applying various line styles to the line structure. In the following specification, after describing the pair data generation unit 310 for synthesizing pair data and the destylization unit 320 capable of performing the destylization model, the content image generated by the destylization model is styled. A description will be made of the styling unit 330 including a model that converts the converted line art.

First, the pair data generation unit 310 of the present invention synthesizes a sketch image from a given line image to generate sketch-line synthesis pair data (S1). In more detail, the pair data generation unit 310 of the present invention includes sketch-line synthesis pair data (hereinafter, “pair data” or “composite pair”) consisting of a sketch image-line image for training the destylization unit 320 to be described later. Data'). To this end, the present invention creates a sketch image inversely from a given clean line image. That is, since the final object of the present invention is to generate a line image from a sketch image, in order to train the model of the present invention, a plurality of pair data consisting of a pair of sketch-lines are required. However, since pair data is difficult to obtain due to problems such as copyright, it is possible to generate pair data for training by generating a sketch image from an already given line image.

In more detail, the pair data generation unit 310 first generates a vector image from a raster line image using vectorization methods. Existing vectorization methods handle raster images with clean lines well, but in many cases raster sketch images cannot be vectorized. On the other hand, the present invention uses a 4th-order Bezier curve capable of modeling a curve shape by three control points.

In more detail, according to an embodiment of the present invention, a vector extracted from a basic curve may be expressed by three control points, and when three control points are given, a corresponding curve may be specified. In more detail, when three control points are defined as c1, c2, and c3, the basic curve B can be expressed as [Equation 3] below.

[Equation 3]

이다. 제어점 c₁ 부터 c₃ 를 변형함으로써, 베지어 커브는 라인의 모양을 직관적으로 변형할 수 있다. 또한, 본 발명은 거친 스케치 스트로크(stroke)를 시뮬레이션하기 위해 제어점의 위치에 노이즈를 추가하여 각 베지어 커브의 파라미터를 교란시킬 수 있다. 또한, 노이즈의 절대적인 크기는 스트로크 길이의 크기에 비례하여 증가하기 때문에, 본 발명은 라인 세그먼트의 길이에 비례하여 섭동(perturbation) 노이즈를 설정할 수 있다. 본 발명은 0의 평균 및

의 표준 편차를 갖는 가우시안 분포로부터 노이즈 ε를 샘플링할 수 있다. 이때, s 는 주어진 깨끗한 스트로크이고, α는 하이퍼파라미터값이다.In the above [Equation 3],

to be. By transforming the control points c ₁ to c ₃ , the Bezier curve can intuitively transform the shape of the line. In addition, the present invention may perturb the parameters of each Bezier curve by adding noise to the position of the control point in order to simulate a rough sketch stroke. Further, since the absolute magnitude of the noise increases in proportion to the magnitude of the stroke length, the present invention can set the perturbation noise in proportion to the length of the line segment. The present invention has an average of 0 and

Noise ε can be sampled from a Gaussian distribution with a standard deviation of. In this case, s is the given clean stroke and α is the hyperparameter value.

In addition, the present invention may randomly divide one long curve into a plurality of short curves, thereby increasing the diversity of synthetic sketches, and imitating the behavior of general artists expressing one long line with a plurality of short lines. To this end, the present invention can evenly divide a line segment into M _c curves and add noise to each curve. Finally, since artists usually reinforce a sketch by drawing a stroke several times, the present invention also repeatedly creates a noise stroke, and the number of iterations can be represented by another hyperparameter M _s . The selection of hyperparameters will be described in the experimental section to be described later.

5 is a diagram illustrating a process of creating a sketch from a clean line image according to the method of creating a synthetic sketch of the present invention.

5, (a) is an input raster line, (b) is a parameterized curve represented by three control points, (c) is the result of control point change, and (d) is a curve with two small strokes. The case is divided into, and (e) is the result of repeatedly generating a sketch stroke.

In more detail, the pair data generation unit 310 according to an embodiment of the present invention converts the input raster line image of FIG. 5(a) into a vector image and converts it into a curve represented by three control points as shown in (b). have. Next, the pair data generation unit 310 may generate another Bezier curve by changing the control points by adding noise to the three control points as shown in (c). Next, the pair data generator 310 may divide one curve into a plurality of small strokes as shown in (d), and add noise to each divided curve as in the method (c). Also, as shown in (e), it is possible to repeatedly generate noise strokes.

In addition, the present invention global augmentations of the composite pair sketch by changing global configurations such as background color, line color and thickness, and global illumination. That is, it is possible to add a texture to the composite pair sketch generated by the above-described method. In addition, the present invention simulates an area made of lines common to sketches. Regarding global lighting, white can be added gradually to account for situations where the light is not evenly distributed. That is, the pair data generation unit 310 of the present invention generates texture augmented composite pair data in which a global setting is added or changed to the composite pair data in addition to the composite pair data obtained by converting a line image into a simple sketch image. can do. The composite pair data may be used to train the line structure extracting unit 323 to be described later, and the texture-expanded composite pair data may be used to train the texture removing unit 322 to be described later.

Next, the destylization unit 320 of the present invention is trained using the paired data described above, extracts a line structure from a rough sketch image, and generates a content image. It includes (S2). In more detail, according to an embodiment of the present invention, the destylization unit 320 may reduce a deviation between sketches and a difference between an actual sketch and a composite sketch by removing a texture. Also, the destylization unit 320 uses a model for normalizing the line width in order to prevent the line width from being deformed due to data expansion by using a line normalizer model.

The destylization unit 320 according to an embodiment of the present invention may include a line normalization unit 321, a texture removal unit 322, and a line structure extraction unit 323. The models of the line normalization unit 321, the texture removal unit 322, and the line structure extraction unit 323 have the same shape as the autoencoder, but there may be differences in model size, use data, and learning loss depending on the role.

6 is for explaining a destylization and stylization model according to an embodiment of the present invention.

The upper part of FIG. 6 is for explaining the training of the destyling unit and the styling unit, respectively, and the lower part is an example of the actual sketch-line conversion result applying the destyling unit and the styling unit, respectively.

As described above, the destylization unit 320 may include a line normalization unit 321, a texture removal unit 322, and a line structure extraction unit 323. Referring to FIG. 6, first, referring to the training part of the destylization unit, the destylization model of the destylization unit 320 of the present invention uses the texture-extended composite pair data, and the texture removal unit 322 is trained. , The line structure extraction unit 313 may be trained using the composite pair data.

In more detail, the line normalization unit 321 prevents the line thickness from being changed during data augmentation of a ground truth line image. Referring to FIG. 6, the line normalization unit 321 may obtain the actual line art y, normalize the line, and provide it to the texture removal unit 322 described later. According to an embodiment of the present invention, the line normalization unit 321 is resized to maintain the thickness of the target line so that the texture removing unit 322 and the line structure extracting unit 323 can obtain accurate results. Can be applied to images. The line normalization model may be smaller than the texture removal model and the line structure extraction model, because the line normalization model considers only a local region. The line normalization unit 321 may be trained with L1 loss using a line image having an extended line width.

Next, the texture removal unit 322 removes a texture such as a background color or a sketch line color from the sketch. The texture removal unit 322 can facilitate the operation of the line structure extraction unit 323 to be described later by removing irrelevant regions and information. Also, the texture removal unit 322 may utilize low-level information.

6, the texture removal unit 322 is trained to remove the texture from the sketch by acquiring the normalized line and the actual rough sketch x from the line normalization unit 321 based on the actual line art (y). I can. In addition, the texture removing unit 322 may be trained with L1 loss using texture augmented synthetic paired data. The actual rough sketch (x) and the actual line art (y) shown in FIG. 6 are actual data that are not paired with each other, and are large amounts of data. That is, x and y are data that are not paired with each other, and are data that can be easily collected because there is no pair of data. In contrast, the composite pair data is sketch-line pair data generated by the present invention. In addition, as shown in FIG. 6, the present invention may add a Generative Adversarial Nets (GAN) loss between the normalized line from the line normalization unit 321 and the actual rough sketch x.

According to an embodiment of the present invention, in order to learn the model of the texture removing unit 322, the present invention may use the composite pair data. The texture removal unit 322 uses the L1 loss and the GAN loss to obtain an accurate result.

The purpose of the texture removal unit 322 may be defined as [Equation 1] below.

[Equation 1]

In the above [Equation 1], N, T, and D _T denote a line normalizing unit 321, a texture removing unit 322, and a discriminator, respectively. In addition, s ₁ , s*, x, and y are respectively a texture-augmented synthetic line image, its paired original clean image, a real sketch image and a real line art. Respectively. More specifically, T in Equation 1 is the learning so as to minimize the value of the objective function D_ _T is a Learning in the direction that maximizes the value of the objective function. According to the present invention, the model to be used for actual texture removing T, D_ _T is to the disturbance, but (due to maximize the value of the objective function as a T in the opposite direction), and finally T learning of the T is minimized Finally can help. In other words, [Equation 1] can be trained to solve difficult T problems easily by questioning problems.

Next, the line structure extraction unit 323 extracts the line structure from the already processed rough image. The line structure means the center of the line regardless of the style of the lines. The line structure extraction unit 323 expresses a line structure using a flat line art image, and the flat line art image is an image composed of only the extracted line structure. The line structure extraction unit 323 may extract the line structure by obtaining input data from the texture removal unit 322.

The line structure extraction unit 323 of the present invention may be trained by input data and synthetic paired data from the texture removing unit 322 as shown in the training part of FIG. 6. At this time, the input data from the texture removal unit 322 may be data from which the texture is removed from the normalized line and the entity sketch (x) as described above, and the GAN loss in the input data from the texture removal unit 322 Can be added. In addition, unlike existing studies, the line structure extractor may be trained with L1 loss using synthetic pair data instead of manually generated data.

In more detail, paired data is used when taking advantage of the L1 loss, which directly informs the correct answer. In addition, since the image from which the texture has been removed by the texture removal unit 322 and the actual line data are not paired with each other, instead of using L1 loss, it is possible to determine how similar the indirectly converted image to the actual line image looks. Use GAN Loss.

For reference, according to the present invention, when the GAN loss learning method does not have a pair for the correct answer, but has each data separately (for example, x, y without a pair as shown in FIG. 6), it is used as much as possible. It is a way to indirectly learn the properties of x or y. In more detail, according to an embodiment of the present invention, there is a model G, so the goal is to make x (real rough sketch) look like y (real line image) (this is expressed as G(x) in the form of a function, and G (x) = y is the goal). In this case, the present invention has a separate model called D, and gives y and G(x) to D to learn to distinguish the two, whereas G allows D to learn not to distinguish between y and G(x). . In this case, D and G perform learning competitively with each other, and if the learning is repeated for a long time, G(x) = y, the goal of the present invention, can be learned. That is, according to the present invention, it is impossible to directly learn G(x) = y because there are (x, y) pairs as data, but it is possible to learn G(x) = y using D.

In addition, the line structure extracting unit 323 may use instance normalization instead of batch normalization. In addition, as shown in FIG. 5, the present invention can add GAN loss. The purpose of the line structure extraction unit 323 may be defined as in [Equation 2] below.

[Equation 2]

In the above [Equation 2], G, D _G , and s ₂ denote a line extraction, a discreminator, and a composite line image, respectively. In more detail, G is learned to reduce the value of the objective expression, and D is learned to increase the value of the objective expression. The content of the objective expression of [Equation 2] is, as in the GAN loss learning method described above, if D distinguishes between the two N(y) and G(T(x)) well, the value increases, and if it is not well distinguished, the value This is going to be smaller. In the same way as the above-described GAN Loss learning method, D learns to distinguish well (to make the objective expression value larger), and G learns not to distinguish well (make the objective expression value smaller). Accordingly, according to the present invention, finally, G (T(x)) generated by G may be similar to actual line data N(y), and more precisely normalized (N) actual line data.

According to an embodiment of the present invention, the line normalization unit 321, the texture removal unit 322, and the line structure extraction unit 323 may be trained at the same time, but in order to use the line normalization unit and the texture removal unit as separate models It is also possible to learn individually and place them together during inference.

7 illustrates the effects of each step of the destylization model.

In more detail, (a) of FIG. 7 is an input sketch, (b) is a result of texture removal, and (c) is a result of line extraction. In more detail, the texture removal unit 322 of the present invention may obtain the same result as (b) by removing a texture other than a line from the input sketch of (a). In addition, the line structure extraction unit 323 of the present invention may extract the line structure and obtain the same result as (c). In addition, (d) of FIG. 7 is an image showing the result of styling by the styling unit 330 described later.

Next, the styling unit 330 of the present invention includes a styling model for adding a style by applying a line style to an image having the extracted line structure (S3). The styling unit 330 of the present invention applies a realistic line style to the flat line image generated by the destyling unit. The present invention converts a flat line image into a stylish line image using an Augmented CycleGAN. In this case, the flat line image is a line image before the stylization of the present invention is applied, and the stylish line image is a line image to which the stylization of the present invention is applied.

As shown in FIG. 6, the styling unit 330 may include a line normalizing unit 331 and a styler 332. The line normalization unit 331 may perform the same role as the line normalization unit 321 of the destylization unit 320 described above. Also, the styler 332 may be trained with L1 loss using a value obtained by normalizing the actual line art y and the actual line art y. In addition, the styler 332 of the styling unit 330 may be trained using an extended CycleGAN learning method using a value obtained by normalizing the actual line art y.

Using the extended CycleGAN, training the styler of the styling unit 330 in the direction of A Х Z _b to B Х Z _a can be expressed as [Equation 4] below.

[Equation 4]

In the above [Equation 4], A, B, S _AB and D _S are a flat line image (sketch), a stylish line image (line image), a styler, and a discreminator, respectively. In addition, Z _b is _a code indicating the style of the line image, and Z _a is _a code indicating the style of the sketch. According to an embodiment of the present invention, similar objectives can be defined by opposite directions, and both directions can be trained simultaneously. Since the line structure can be easily extracted from the line image of a clean style using the above-described line normalization unit 321, the present invention performs supervised loss for semi-supervised learning. It can be added as in [Equation 5].

[Equation 5]

According to the present invention, it is possible to prevent steganography, which may occur when the map loss part is training the cycle coherence loss model. Even if the extended CycleGAN fails to directly convert the sketch image into a stylish line image, The extended CycleGAN can successfully train styling flat line structures with the aid of the half-map loss described above.

Referring to the application of the lower part of FIG. 6, a style guided latent code or random noise is added to the flat line art image output by the line structure extraction unit 323, and the styling unit 330 The styler 332 of) may output a multi-modal line art image. In connection, when the style image exists according to the present invention, but using an encoder used in from style image pull the Z _b (style of the line image code), and when there can be extracted the Z _b on the random sampling . For example, if Z _b is represented by 10 numbers, 10 numbers can be extracted by random sampling from a Gaussian distribution.

The style-based sketch-line conversion method of the present invention includes a destylization model for extracting content from a sketch image and a stylization model for applying various line styles to the extracted content. Previous techniques used unsupervised image-image conversion, and there was a problem that style conversion could not distinguish noise lines and details. However, the present invention can preserve important details of content using supervised learning while successfully removing noise. The destylization model includes a line normalization unit, a texture removal unit, and a line structure extraction unit, and is trained using sketch-line pair data synthesized by the present invention. After extracting the line structure, the present invention converts a flat line image into a stylish line image using the extended CycleGAN.

Hereinafter, an experiment result of comparing the sketch-line conversion method based on the style of the present invention with the existing research results will be described. However, the following experimental results are only an embodiment of the present invention, and details can be changed without departing from the core features of the invention.

First, the design of the experiment will be briefly described. As data for experimenting the sketch-line conversion method of the present invention, the Danbooru2017 data set (G. B. A. G. Anonymous, the Danbooru community. Danbooru 2018: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset. January 2019.) was used. This data set consists of over 3.3 million animated images with 99,700,000 tag annotations. Existing Inker models using 280 pairs of sketches and line images (E. Simo-Serra, S. Iizuka, and H. Ishikawa. Real-Time Data Driven Interactive Rough Sketch Inking. ACM Transactions on Graphics (SIGGRAPH), 37(4) ), 2018.), we manually collected 300 sketch and line images, respectively, from images tagged with sketch and line art. When collecting line images, it was considered that each line image represents a distinct style according to thickness, brightness, and curvature. In addition, 900 composite sketches were created using 30 line images to generate composite pair data according to the present invention. For training, a patch extracted from each image expanded by the method described in the Inker model described above was used.

₁ = 0.5,

₂ = 0.999 인 ADAM 최적화 도구(D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5)를 사용했다. 또한, 상술한 [수학식 1] 내지 [수학식 5] 에서

, γ₁, γ₂를 각각 10^-3, 1.0, 0.025로 설정 하였다. 합성 스케치 구축에 사용 된 하이퍼 파라미터에 대해 α∈{30, 40, 50}과

= 3 인 포아송 분포로부터의 무작위로 추출한 M_c 및 M_s 를 값을 값을 사용했다. 또한, 본 실험에서는 채색된 영역 또는 채워진 영역의 비율을 이미지 영역의 20 %로 설정했다.In this experiment, the line normalization unit, the texture removal unit, and the line structure extraction unit of the present invention were trained by repeating 20,000 times with one mini-batch size, and training the styling unit by repeating 60,000 times. Also, the learning rate is 0.0001,

₁ = 0.5,

₂ = 0.999 ADAM optimization tool (DP Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5) was used. In addition, in [Equation 1] to [Equation 5] described above

, γ ₁ and γ ₂ were set to 10 ^-3 , 1.0, and 0.025, respectively. For the hyperparameters used to build the composite sketch, α∈{30, 40, 50} and

Values of M _c and M _s extracted at random from the Poisson distribution of = 3 were used. In addition, in this experiment, the proportion of the colored area or the filled area was set to 20% of the image area.

Next, the experiment will be introduced in more detail. In this experiment, extensive controlled experiments were performed to show the effects of the above-described destylization model and stylization model.

8 is an example for explaining the effect of the styling model of the present invention.

8 is a result of using various line styles generated by the conversion method of the present invention. In more detail, (a) of FIG. 8 is an input sketch, (b) is a result image stylized as a thick line segment, and (c) is a result image stylized as a thin segment. Referring to FIG. 8, it can be seen that the resulting image styled according to the line style is generated differently. In other words, it can be seen that according to the present invention, different line styles are successfully reflected while maintaining important details of a given sketch image. In addition, it can be seen that the map loss equation of [Equation 5] prevents the content removal shown in the existing method.

Next, an experimental result comparing the sketch-line conversion method of the present invention and the conventional conversion method will be introduced. In this experiment, the conversion method of the present invention and the result of the existing model will be compared. At this time, the model to be compared is WCT (Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang. Universal style transfer via feature transforms. In NIPS, 2017. ), Inker (E. Simo-Serra, S. Iizuka, and H. Ishikawa. Real-Time DataDriven Interactive Rough Sketch Inking. ACM Transactions on Graphics (SIGGRAPH), 37(4), 2018.), Extended CycleGAN (A. Almahairi, S. Rajeshwar, A. Sordoni, P. Bachman, and AC Courville.Augmented cyclegan: Learning many-tomany mappings from unpaired data.In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmδssan, Stockholm, Sweden , July 10-15, 2018, pages 195-204, 2018.) It is assumed that it is a model.

First, to evaluate the model's performance, 20 users were asked to select the most visually attractive image based on how each model preserves the original sketch content and conveys the style as a refined line art. For the test images, 40 sketch images were randomly selected from sketches not used during training in Danbooru 2017 described above. In this evaluation, two user studies were conducted. First, the first was requested to evaluate the overall performance of the sketch-line conversion method for WCT, Inker, extended CycleGAN, and the present invention. Second, we asked to evaluate Inker's results and the destylization model of the present invention. For the results of WCT and Inker, we used the results generated at the project site, and for the results of AugCycleGAN, we implemented ourselves and trained until the results converge.

Table 1 shows the results of user studies of this model using WCT, Inker, AugCycleGAN and WCT. Here, each model represents the latest model of the sketch simplification, image-image and style transfer domain model respectively. As shown in the table, it can be seen that the user's most preferred method is the present invention. The model of the present invention tends to better preserve the original contents of the input sketch compared to MUNIT and AugCycleGAN. In the case of Inker, the structure of the line is well maintained, but the user prefers the result of the present invention because the line style and the important factors that determine the atmosphere of the line are not considered.

Positive answer rate Dominant image number WCT 0% 0 (0%) Inker 31.1% 9(22.5%) Extended CycleGAN 26.8% 18 (20%) The present invention 42.1% 19 (47.5%)

In addition, the results of other user studies for evaluating the performance of the destylization model of the present invention are presented in [Table 2]. In this experiment, the results of the destylization model were compared with Inker. In this study, in order to investigate the quality of the line structure, not the line style, the line width was normalized using the line normalization unit so that the user could not distinguish the model according to the line thickness. As can be seen from [Table 2], it can be seen that the model of the present invention exhibits the best performance among sketch simplification methods without actual sketch line data.

Positive answer rate Dominant image number Inker 40.6% 15 (37.5%) The present invention 59.4% 25 (62.5%)

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium may continue to store a program executable by a computer, or may be stored for execution or download. In addition, the medium may be a variety of recording means or storage means in a form in which a single piece of hardware or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, and may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks, and And ROM, RAM, flash memory, and the like may be configured to store program instructions. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or storage medium managed by a server.

In the above, the present invention has been described by specific matters such as specific elements and limited embodiments and drawings, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. Anyone with ordinary knowledge in the technical field to which the invention belongs can make various modifications and changes from these descriptions.

Accordingly, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all ranges equivalent to or equivalently changed from the claims to be described later as well as the claims to be described later are the scope of the spirit of the present invention. It will be said to belong to.

Claims (17)

A pair data generator for synthesizing a sketch image from a given line image to generate sketch-line combination pair data;
A destylization unit for extracting a line structure by removing texture from a rough sketch image using the destylization model trained by the paired data;
A styling unit for adding a style by applying a line style to the extracted line structure image using a styling model;
Including,
The pair data generation unit,
Generates a noise stroke based on a plurality of control points obtained by converting a raster line image into a vector image,
Sketch-to-line conversion system based on style. The method of claim 1,
The pair data generation unit,
Noise is generated by creating a vector image from a raster line image and defining a curve using a plurality of control points, adding noise to the plurality of control points to create a new curve, and adding noise by dividing the curve into a plurality of small curves. A style-based sketch-to-line conversion system that creates strokes. The method of claim 1,
The destylization unit includes a texture removal unit to remove a texture and a line structure extraction unit to extract a line structure, based on a style sketch-line conversion system. The method of claim 3,
The destylization unit further includes a line normalization unit that normalizes the line width so that the line width is not deformed due to data augmentation of the ground truth line image. Sketch-line conversion based on style system. The method of claim 3,
The texture removal unit removes the background color or the sketch line color from the sketch, a style-based sketch-line conversion system. The method of claim 3,
The texture removal unit is trained using texture augmented composite pair data, and the line structure extractor is trained using composite pair data. The method of claim 1,
The line structure is a style-based sketch-line conversion system, which is the center of a line unrelated to the style of the rough sketch. A destylization step of extracting a line structure by removing texture from a rough sketch image using a destylization model; And
A style that applies a line style to an image composed of the extracted line structure using a styling model to add a style, and generates a noise stroke based on a plurality of control points obtained by converting a raster line image into a vector image. The stage of fire; Sketch-line conversion method based on the style including a. The method of claim 8,
The destylization step,
A texture removal step of removing texture; And
A line structure extraction step of extracting a line structure; including, a sketch-line conversion method based on a style. The method of claim 9,
The destyling step further comprises a line normalization step of normalizing the line width so that the line width is not deformed due to data augmentation of the ground truth line image. -Line conversion method. The method of claim 9,
Wherein the texture removal step is trained using texture augmented composite pair data, and the line structure extraction step is trained using composite pair data. The method of claim 9,
The texture removal step removes a background color or a sketch line color from the sketch. A pair data generation step of synthesizing a sketch image from a given line image to generate sketch-line combination pair data;
A destylization step of extracting a line structure by removing texture from a rough sketch image using a destylization model trained by the paired data; And
A styling step of adding a style by applying a line style to the extracted line structure image using a styling model; Including,
The pair data generation step,
Generates a noise stroke based on a plurality of control points obtained by converting a raster line image into a vector image,
Style-based sketch-to-line conversion method. The method of claim 13,
The pair data generation step,
Noise is generated by creating a vector image from a raster line image and defining a curve using a plurality of control points, adding noise to the plurality of control points to create a new curve, and adding noise by dividing the curve into a plurality of small curves. A style-based sketch-to-line conversion method that creates a stroke. The method of claim 13,
The destylization step,
A texture removal step of removing texture; And
A line structure extraction step of extracting a line structure; including, a sketch-line conversion method based on a style. The method of claim 13,
The destyling step further comprises a line normalization step of normalizing the line width so that the line width is not deformed due to data augmentation of the ground truth line image. -Line conversion method. A computer program recorded on a computer-readable recording medium for executing the method according to any one of claims 8 to 16.

Images