KR20200064591A - Webtoons color customizing programs and applications of deep learning - Google Patents

Abstract

The present invention relates to a deep learning based webtoon coloring customization program and an application therefor. The deep learning based webtoon coloring customization program can execute, in a computer having an input means for inputting data and a processing means for processing the input data, the steps of: (a) receiving a webtoon image from a user by the input means; (b) generating, by the processing means, a coloring model learned through a deep leaning based algorithm by using black and white sketch image and original color image as input values; (c) inferring, by the processing means, colors based on the coloring model to automatically color the black and white sketch image of a user; (d) selecting a partial colored area by the user; and (e) selecting a combined coloring pattern by the user.

Description

Deep learning based webtoon color customization program and application {WEBTOONS COLOR CUSTOMIZING PROGRAMS AND APPLICATIONS OF DEEP LEARNING}

The present invention relates to a program and application for learning a user's sketch style based on deep learning to color a webtoon, and in particular, to a program and application capable of providing various customized coloring patterns by combining a user's sketch pattern.

Webtoon is a compound word of'web', which means the Internet, and'cartoon', which means cartoon. It is an internet cartoon produced using various multimedia effects. Webtoons are produced in the form of creators sketching and coloring manually. Unlike cartoons that were generally published in black and white drawings, webtoons have the characteristic of being provided to consumers as color images that have been colored.

When creating a webtoon, if the creator manually colors each one, there is a problem that it takes a lot of time to produce the webtoon. Accordingly, the creators of webtoons have also placed auxiliary artists to reduce the coloring time. When creating a webtoon, artistic talent is required to select a suitable color for the situation in a black and white sketch, and to create a vivid cartoon image that properly combines light and shadow. This process of coloring is not an easy task for ordinary people, and it is not easy to find an assistant writer who colors the webtoon as he wants. Therefore, if an application for automatically coloring the sketch image of the creator is provided, time and manpower consumption can be reduced compared to the existing manual coloring work.

Conventionally, a sketch coloring application such as Paintchainer exists, but the application is a program learned based on illustrations with various colors, and has a limitation that the result is different from the creator's characteristic coloring style. Each webtoon has a unique coloring style for each artist, so there is no reason to use it when the program that automatically paints is not colored in the style you want. Therefore, there is a need for a technique that automatically paints and provides sketch images similar to the artist's individual coloring style.

On the other hand, it is common for creators to adhere to the sketch style they have been accustomed to for a long time. This is because even if you imitate the coloring style of other creators, they are immature and unnatural or awkward. For this reason, creators may have difficulty creating new works outside their own style. Accordingly, the present applicant has devised a color customizing program and application that can automatically color by reflecting the sketch style of the creator, and partially provide a new coloring style in which other people's sketch style is combined by deep learning.

Patent Publication No. 10-2018-0066440

The present invention seeks to provide a deep learning based webtoon color customizing program and application that provides an automatically colored image similar to a user's coloring style.

In addition, the present invention is to provide a coloring learning program and application of deep learning based webtoon that can present a variety of coloring results by combining the coloring style of others with their coloring style.

In order to achieve the above object, the present invention, a computer having an input means for inputting data and a processing means for processing input data, (a) the input means receiving a webtoon image from a user; (b) the processing means generating a coloring model trained by a deep learning based algorithm using the black and white sketch image and the original color image of the webtoon image as input values; (c) automatically processing the user's black and white sketch image by inferring color based on the coloring model; (d) the user selecting a partial colored area; And (e) a user selecting a combined coloring pattern.

According to the present invention, as a user learns a coloring pattern for each user, automatic coloring is performed using a characteristic coloring style unique to the creator. This has the effect of applying to a new webtoon of the same user and quickly obtaining an image reflecting the user's coloring method. Accordingly, the present invention can quickly apply its coloring style to other webtoon works of the same webtoon artist.

In addition, according to the present invention, a combination of coloring may provide a sketch style that is difficult to implement by hand, and a variety of coloring results in which one's sketch style and another person's sketch style are combined can be selected, and revenue is shared by sharing the coloring pattern. Creation is possible. Accordingly, it is possible to greatly contribute to the motivation, freedom and convenience of webtoon creation to the creator.

1 shows a system flowchart of a webtoon automatic coloring program according to an embodiment of the present invention.
2 shows an example in which a frame of a webtoon is extracted by performing a black and white sketch conversion step according to an embodiment of the present invention.
3 shows an example in which the original color image, which is the frame in which the black and white sketch conversion step is extracted, is converted into a black and white sketch image.
4 shows a method of learning coloring with the GAN algorithm in the learning stage.
5 illustrates a system flow diagram of a color customization program according to an embodiment of the present invention.
FIG. 6 shows an embodiment of selecting a region of a cut in the step of inputting a coloring pattern in the embodiment of FIG. 5.
FIG. 7 shows a system flow diagram of the step of determining a coloring pattern in the embodiment of FIG. 5.
8 shows an example of an algorithm for combining color patterns.

Hereinafter, the present invention will be described in detail with reference to the contents described in the accompanying drawings. However, the present invention is not limited or limited by the exemplary embodiments. The same reference numerals in each drawing denote members that perform substantially the same function.

The objects and effects of the present invention may be naturally understood or more apparent by the following description, and the objects and effects of the present invention are not limited only by the following description. In addition, in the description of the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

1 shows a system flowchart of a webtoon automatic coloring program according to an embodiment of the present invention. The webtoon automatic coloring program is a computer equipped with an input means for inputting data, a processing means for processing the input data, and an output means for outputting the processed data. The (b) step (S30) to be generated and the (c) step (S50) to be automatically colored may be executed.

The computer may include input/output devices such as a keyboard and a mouse as input means, and a processor as a processing means. The webtoon automatic coloring program according to the present embodiment may be stored in a downloadable web or media and distributed to a user, and may be downloaded on a user's computer and executed in each step.

The webtoon automatic coloring program according to the present embodiment is a system for automatically colorizing a black and white sketch webtoon based on deep learning, and performs automatic coloring using a learning model. In addition, by creating a learning model for each webtoon writer who is the user of the present invention, it is possible to apply unique colors and colors of each artist to a new black and white sketch. 1 shows the overall system configuration and flow chart of a webtoon automatic coloring program.

Referring to FIG. 1, step (a) (S10) may be a step in which a computer input means receives a webtoon image from a user. The webtoon image referred to in this specification does not only mean a cartoon that is serialized online, but may refer to an image that processes a drawing on a computer equipped with a processor. That is, the webtoon image of the present specification may include a sketched or colored image using an online web tool, and an online web for printing a cartoon image, a continuation image for animation production, an image for illustration production, and an offline comic book. It may be a sketched image using a tool. Also, the webtoon image may include an image drawn through the user's smartphone or tablet and an image taken using the user's smartphone, tablet, or other device's camera.

Step (a) (S10) means a step of receiving a webtoon image, which is a picture image of the original sketched by the user. Here, the original webtoon image directly sketched by the user may be not only a sketch image created on a computer, but also a computer image file that scans the sketched image offline or a photo image taken using a camera of a user's smartphone or tablet or other device. It is okay. In step (a) (S10), the original webtoon image of the artist produced on-offline is input through the computer's input means.

In this embodiment, step (a) (S10) may receive an original color image of the webtoon, which has been colored, from the user. The webtoon image input in step (a) (S10) may be a black and white or original color image. However, in step (a) (S10), the original color image of at least one webtoon must be input to learn the coloring style for each user during learning. In the present embodiment, step (a) (S10) describes a case in which the original color image is first input from the user for initial learning. After creating a coloring model according to learning, step (a) (S10) receives a black and white sketch image as a webtoon image from the user.

Step (b) (S30) may be a step in which the computer's processing means generates a coloring model trained by a deep learning-based algorithm using black and white sketch images of webtoon images and original color images as input values. The webtoon automatic coloring program according to this embodiment may be divided into a function of generating a learning model based on a set of learning data and a function of inferring color based on the generated learning model. The function of generating a learning model based on the learning data set is performed in step (b) (S30), and the function of inferring color can be performed in step (c) (S50), which will be described later.

(B) Step (S30) may include a step of determining whether to re-learn (S301), a black and white sketch conversion step (S303) and a learning step (S305).

The re-learning determination step S301 may be a step of performing re-learning or new learning according to a user's selection. In re-learning or not (S301), when performing new learning, a user's webtoon image is used as a new data set to learn with a deep learning-based algorithm to generate a new coloring model, and when re-learning, an existing coloring model is used. Let's learn the user's webtoon image.

The step of determining whether to re-learn may be implemented as a function button of re-learning or new learning to proceed according to a user's selection. If the user wants continuous learning of the series to be automatically colored, he/she can select re-learning, and in this case, the coloring model of step (b) (S30) previously performed is updated. On the other hand, if the user needs to apply a different coloring style such as a new webtoon series, the user can select a new learning. When the user selects a new learning, in step (a) (S10), an original color image is received and a new coloring model generation process is performed. Hereinafter, a detailed function of step (b) (S30) for generating a coloring model through deep learning is as follows.

The black and white sketch conversion step (S303) means a step of extracting individual frames of the original color image and converting them into a black and white sketch image. (a) In the webtoon image input in step S10, individual frames must be extracted according to the learning dataset format. In the case of webtoons, it is characterized by a vertically long form, and individual frames exist in various positions. 2 shows an example in which a black and white sketch conversion step (S303) is performed and the webtoon frame is extracted.

In the black and white sketch conversion step (S303), methods such as horizontal and vertical scan-based masking, labeling, edge search, morphological operation, stochastic hub conversion, convex hull, and bounding box may be applied to extract individual frames. Deep learning requires a large amount of learning data, and it is inefficient and difficult to manually extract individual frames within webtoons manually to satisfy these needs. For this reason, the black and white sketch conversion step (S303) automatically extracts individual frames in the webtoon using the frame extraction method. Referring to FIG. 2, a webtoon image input in step (a) (S10) from a left side, and a black and white sketch conversion step (S303) is performed, so that the image extracted from the frame and the image resulting from the frame extraction can be confirmed. As a result of the frame extraction, it is possible to normalize the frame size of each webtoon scene from the original webtoon image.

In the black and white sketch conversion step (S303), the size of the frame is normalized and extracted, and then black and white sketching is applied. The training data set for automatic coloring requires a pair of black and white sketch images and original color images. The two images are required because the deep learning application algorithm of the present embodiment, which will be described later, is based on unsupervised learning. Since the learning dataset is difficult to construct, an initial original webtoon image can be sketched in black and white.

The black-and-white sketch conversion step S303 may convert the original color image into a black-and-white sketch image by applying a method such as a difference between a grayscale image of the original color image and an image that has been expanded. Alternatively, it can be converted using a deep learning-based algorithm that inputs a 3-channel color image as an input and outputs a 1-channel black and white sketch image as an output. 3 illustrates an example in which the original color image, which is the frame in which the black and white sketch conversion step S303 is extracted, is converted into a black and white sketch image.

Then, step (b) (S30) performs a learning step (S305) to generate a coloring model.

The learning step S305 may mean a step of learning the original color image and the black and white sketch image as an unsupervised learning model. In this embodiment, the learning step (S305) may learn a black and white sketch image and an original color image as an unsupervised learning generation model. Machine learning is largely divided into supercised learning and unsupervised learning. Supervised learning, which has been the mainstream of machine learning to date, is a method of intensively learning pairs of data and labels (names). As an example, the computer continuously labels the object X and the name Y by inputting a label of'dog' or cat'. Knowing such a large number of (X, Y) functions, it is possible to produce a proper prediction Y for an unknown object X, which is unknown at all. The machine learning technology based on supervised learning is used for face recognition, object recognition, and voice recognition used in Google and Facebook. However, in supervised learning, there is a limit in that learning is possible only when someone individually informs the correct label of each object. In order to realize the ultimate artificial intelligence, even if someone does not teach the correct answer, the artificial intelligence must have the ability to grasp the characteristics of the object by itself. This method of obtaining knowledge from the data itself without a label is called unsupervised learning.

In this embodiment, the learning step S305 generates an unsupervised learning-based coloring model using a method of supervised learning in a situation in which there is no correct answer. In this embodiment, the learning step (S305) may be applied to the GAN (Generative Adversarial Network) as an unsupervised learning generation model.

GAN is an algorithm that learns through competition with counterfeits. In the learning step S305, a deep learning algorithm equipped with a G model approximating the probability distribution of the training data and a D model for determining the original color image, the color of the G model generated by approximating the probability distribution of the original color image An image can be trained to determine that the D model is the original color image. In this embodiment, the G model and the D model may be configured as a convolutional neural network.

The G model can be understood as a generator. The D model can be understood as a discriminator. The G model strives to make real counterfeits, while the D model seeks to develop each other by trying to distinguish the real from the fake, like a discriminator.

4 illustrates a method of learning coloring using the GAN algorithm in the learning step (S305). The network structure of the G model can use a U-net structure that connects the encoder layer to the corresponding layer of the decoder. The G model decodes the low-level information of the black and white sketch image and connects it to the encoder layer to get a colorful image. Here, the encoder layer includes sketch edge information, and the decoder layer mainly includes learned color information.

The D model uses only the encoder unit compared to the G model, and learns to maximize discrimination accuracy with the goal of determining whether the generated color image or the original color image is real or fake. Automatic coloring in black-and-white sketch images can be a more difficult problem than normal grayscale coloring. This is because black and white sketch images have much less information than grayscale images. In order to solve the above problem and improve performance, various loss functions are required in the G model.

In this embodiment, the G model may define a softmax cross entropy between the generated color image and the real label (0) as a loss function 1. The G model can define the mean absolute error (MSE) of the generated color image and the original color image as a loss function 2. The G model may define the MSE of edge feature information for the last generated color image and the original color image as a loss function 3. Each loss function can set a weight, and the sum of all loss functions can be defined as the final loss function. The correlation between the black and white sketch image and the color image is learned to minimize the loss function.

The loss function of the D model can be defined as the sum of the softmax cross entropy of the original color image and the real label (0) and the softmax cross entropy of the generated color image and the fake label (1). The D model is trained so that the defined loss function is minimized. A color model is created by learning like this. When learning progresses as much as a user-defined epoch, a color model is generated, and a color model can be generated for each generation.

Step (c) (S50) may be a step in which the computer's processing means infers the color based on the coloring model and automatically colors the user's sketch image. The automatically colored image can be displayed on the user's computer through the computer's output means. In the color model, the color modeling based on the learning model first loads the color model desired by the user. In addition, when the user inputs a new black and white sketch image, color can be inferred based on the imported color model to perform automatic coloring.

Step (c) (S50) calls the coloring model according to the user and can automatically color the sketch image by inferring the color reflecting the sketch characteristics of the user. In this case, in step (b) (S50), when it is determined that two or more users show similar coloring patterns, a coloring model may be generated by learning together a plurality of webtoon images input by two or more users.

In another embodiment, the webtoon automatic coloring program may be provided as an application. The webtoon automatic coloring application is a smart phone, a tablet, or a laptop equipped with an input means for inputting data, a processing means for processing input data, and an output means for outputting processed data, wherein the input means inputs a webtoon image from a user Receiving first function; A second function for the processing means to generate a coloring model trained by a deep learning-based algorithm using the black and white sketch image and the original color image of the webtoon image as input values; And the processing means may execute a third function for automatically coloring the sketch image of the user by inferring color based on the coloring model.

In this embodiment, the first function is the above-mentioned step (a) (S10), the second function is the above-described step (b) (S30), and the third function is the above-described step (c) (S50). . Recently, with the advancement of IT technology, large-scale tablet PCs for drawing for webtoon writers have been spreading. In the past, webtoon writers who had to sketch directly on paper with a pen can use the tablet PC's drawing application to work with sketches, corrections, and coloring with the same precision as a real pen, as well as various blurring, brushes, etc. The effect is easy to implement, so it is preferred over traditional paper or PC work. Accordingly, the webtoon automatic coloring program according to the present embodiment may be provided as an application stored in a medium in order to be distributed to a smartphone, tablet, or notebook.

5 illustrates a system flow diagram of a color customization program according to an embodiment of the present invention. FIG. 6 shows an embodiment of selecting a region of a cut in the step of inputting a coloring pattern in the embodiment of FIG. 5. FIG. 7 shows a system flow diagram of the step of determining a coloring pattern in the embodiment of FIG. 5. 8 shows an example of an algorithm for combining color patterns.

6 to 8, the coloring customization program according to the embodiment of the present invention is applied by adding a coloring (color) pattern designated by the manufacturer's intention after automatic coloring is completed in a method of coloring cartoons or images, etc. To change the color of the image. The coloring pattern can be assigned to reflect a specific color by automatic coloring, and the designation of the coloring (color) pattern is illustrated in FIG. 7.

Referring to FIG. 7, a label may be formed to designate an entire image, adjust only a portion of a horizontal area, adjust only a portion of a vertical area, or adjust only a region of a user-specified cut. As an example, when a user designates a selection area for each cut, an area is selected in the corresponding image and added to the area to perform automatic coloring, and the corresponding cut may be a single sheet or multiple sheets. Also, it is basically a continuous cut, but sometimes it may be designated non-consecutively.

The user's pattern may be a color, and may be a specific shape such as a specific object, a specific animal, a human body shape, fashion clothing, and sundries. In particular, in the case of a person, the face, body, arms, clothes, and other miscellaneous goods are recognized through AI, and the area is designated and colored. In the case of automatic coloring, the result value is displayed as the image tone of the person designated by the user. Can be represented.

The user's pattern can be designated and created according to the user's intention, and saved as A, B, C, D, E, so that the user can use the pattern as intended, and the user (original user) can share the pattern You can let other users use the pattern. At this time, other users can share the income when they pay the corresponding cost to the original user who created the pattern or obtain income using the pattern. At this time, the profit is divided according to the ratio. At this time, the ratio can be determined by the original user and other users, authoring tools or platforms (if any).

The method of using automatic coloring in FIG. 8 is a method of selecting a basic pattern of automatic coloring before performing automatic coloring, and applying a coloring pattern to vary the result of coloring. At this time, the type of automatic coloring and the type of coloring pattern can be made by the user.

Although the present invention has been described in detail through exemplary embodiments above, those skilled in the art to which the present invention pertains understand that various modifications are possible within the limits of the embodiments described above without departing from the scope of the present invention. will be. Therefore, the scope of rights of the present invention should not be limited to the described embodiments, but should be determined by any modified or modified form derived from the claims and equivalent concepts as well as the claims described below.

Claims (5)

To a computer equipped with input means for inputting data and processing means for processing inputted data,
(a) the input means receiving a webtoon image from a user;
(b) the processing means generating a coloring model trained by a deep learning-based algorithm using black and white sketch images of the webtoon image and original color images as input values;
(c) automatically processing the user's black and white sketch image by inferring color based on the coloring model;
(d) the user selecting a partial colored area; And
(e) Color customizing program of webtoons stored in the medium to execute the step of selecting the combined coloring pattern by the user.
According to claim 1,
Step (d) is,
Webtoon color customization program stored in the media, characterized in that a label is provided to designate the entire image, specify the horizontal area of the image, specify the vertical area of the image, or specify the selected area of the image as a closed loop.
According to claim 1,
Step (e) is,
Webtoon's coloring customization program stored in media, characterized by including the pattern of colors, colors, specific objects, specific animals, human body shapes, fashion clothes, and miscellaneous goods.
According to claim 1,
Step (e) is,
Generating a coloring pattern by combining the learned user's sketch proportionately with other coloring sketches;
Classifying and storing the coloring pattern; And
Coloring customization program of the webtoon stored in the medium, characterized in that it comprises the step of sharing the classified coloring pattern.
In a smartphone, tablet, or laptop equipped with input means for inputting data, and processing means for processing input data,
A first function in which the input means receives a webtoon image from a user;
A second function for the processing means to generate a coloring model trained by a deep learning-based algorithm using the black and white sketch image and the original color image of the webtoon image as input values;
A third function in which the processing means infers a color based on the coloring model to automatically color a sketch image of the user;
A fourth function in which the user selects a partial colored area; And
A webtoon auto-coloring application stored in the medium to execute a fifth function in which the user selects a combined coloring pattern.

Images