KR102449790B1 - Apparatus and method for automatically coloring sketch images - Google Patents

Abstract

Provided are an apparatus and a method for automatically coloring sketch images that automatically can color the sketch images of webtoons more accurately and quickly. The apparatus includes: a first region dividing unit dividing a sketch image, which is to be colored, in first unit areas using a first region division technique; a second region dividing unit dividing the sketch image into second unit regions to which classes are mapped by using a second region division technique while the classes are mapped to pixels of the sketch image respectively and the sketch image is divided into the second unit regions based on the class mapped to each of the pixels; a merging unit generating one or more colored unit regions to each of which a class is mapped by using the first unit regions and the second unit regions; and a coloring unit coloring the sketch image using a color corresponding to the class mapped to each of the coloring unit regions.

Description

Apparatus and method for automatically coloring sketch images}

The present invention relates to an apparatus and method for automatically coloring a sketch image, and more particularly, to an apparatus and method for automatically coloring a sketch image of a webtoon.

Webtoon is a compound word of web, meaning internet, and cartoon, meaning cartoons, and refers to Internet comics produced by mobilizing various multimedia effects.

Webtoons are produced in the form of sketches by creators and manual coloring. In general, unlike cartoons that were published as black and white drawings, webtoons have the characteristic of being provided to consumers as color images with coloring completed.

There is a problem in that it takes a lot of time to make a webtoon if the creator manually colorizes each one when making a webtoon. For this reason, the webtoon creator also has an assistant artist to reduce the coloring time. This coloring process is not an easy task for ordinary people, and it is not easy to find an assistant artist who can color the webtoon as desired.

As such, coloring webtoons is a task that takes a lot of time and money, so studies are being made to automate it.

The matters described in the background art above are intended to help the understanding of the background of the invention, and may include matters that are not disclosed 　 prior art.

Prior Art 1: Republic of Korea Patent No. 10-2335702 (Domain Adaptive Semantic Image Segmentation Apparatus and Method) Prior art 2: Republic of Korea Patent Registration No. 10-2216749 (method, device and computer program for completing coloring of target image)

The present invention has been proposed in view of the above-described conventional circumstances, and an object of the present invention is to provide an apparatus and method for automatically coloring a sketch image that automatically and more accurately and quickly processes coloring on a sketch image of a webtoon.

In order to achieve the above object, an apparatus for automatically coloring a sketch image according to a preferred embodiment of the present invention includes: a first region dividing unit that divides a sketch image to be colored into first unit regions using a first region dividing technique; The sketch image is divided into class-mapped second unit regions using a second region division technique, a class is mapped for each pixel of the sketch image, and the sketch image is generated based on the class mapped for each pixel. a second region dividing unit dividing into two unit regions; a merging unit generating one or more color unit regions to which classes are mapped by using the first unit regions and the second unit regions; and a coloring unit that colors the sketch image by using a color corresponding to a class mapped to each of the coloring unit areas.

The second region divider may verify and automatically adjust the class mapped to each pixel based on the inclusion relationship between the classes.

The merging unit may generate one or more colored unit areas by mapping the class for each pixel for which the verification and automatic adjustment have been completed, to each pixel of the first unit areas.

The second region divider may verify the correctness of the class mapping by gradually moving from the upper class hierarchy to the lower class hierarchy.

The second region divider, for each pixel, obtains a probability that each of the classes in the current class hierarchy is mapped, and selects the class having the highest probability among the classes included in the class having the highest probability in the upper class hierarchy. It can be determined as a class in the current class hierarchy.

The second region dividing unit is configured to select the class having the highest probability if the class having the highest probability among the classes in the current class hierarchy is not included in the class having the highest probability in the previously determined upper class hierarchy. Without determining the class of the current class hierarchy, the class having the highest probability among the remaining classes except for the class having the highest probability may be determined as the class of the current class hierarchy.

The first region segmentation technique may be an edge-based image region segmentation algorithm.

The coloring unit compares a plurality of divided regions and a reference colored image including color information of each of the plurality of divided regions with the sketch image, extracts color information corresponding to each coloring unit region of the sketch image, and extracts the extracted color The sketch image may be colored based on the information.

The coloring unit may determine region reliability for each coloring unit region of the sketch image, and color the coloring unit region having the determined region reliability lower than a predetermined reference value with a color corresponding to other coloring unit regions. have.

The coloring unit may color the coloring unit area having the area reliability lower than the reference value with a color corresponding to the coloring unit area having the same or similar shape and class among the coloring unit areas of the sketch image.

The coloring unit may perform color correction on the corresponding coloring unit area according to a color correction command from the user terminal.

A preferred embodiment of the present invention may further include a feedback reflecting unit that receives user feedback information for adjusting the class mapped for each colored unit area from the user terminal and transmits it to the merging unit.

A preferred embodiment of the present invention may further include a feedback reflecting unit that directly adjusts a class for a color unit area to be adjusted based on user feedback information from a user terminal among the classes mapped for each color unit area.

Meanwhile, a method for automatically coloring a sketch image according to a preferred embodiment of the present invention includes a first step of dividing a sketch image to be colored into first unit regions using a first region division technique; The sketch image is divided into class-mapped second unit regions using a second region division technique, a class is mapped for each pixel of the sketch image, and the sketch image is generated based on the class mapped for each pixel. a second step of dividing into two unit regions; a third step of generating one or more color unit regions to which classes are mapped by using the first unit regions and the second unit regions; and a fourth step of coloring the sketch image using a color corresponding to a class mapped to each of the coloring unit areas.

According to the present invention of this configuration, a sketch image to be colored is divided into an edge-based image segmentation and a semantic image segmentation, respectively, and a class is mapped by the semantic image segmentation technique to the edge-based segmented area to correspond to the corresponding area. Since colors can be automatically colored, the time required for coloring sketch images can be reduced. Thereby, there is an effect that can reduce the time required to complete the webtoon.

In particular, the present invention can automatically perform class verification and adjustment of class-mapped partitions based on the class hierarchy in which the inclusion relationship between classes is defined. It can relieve loneliness. For this reason, there is an effect of providing convenience in use.

1 is a system configuration diagram in which an automatic sketch image coloring apparatus according to an embodiment of the present invention is employed.
FIG. 2 is an internal configuration diagram of the server shown in FIG. 1 .
FIG. 3 is an internal configuration diagram of the control unit shown in FIG. 2 .
FIG. 4 is a diagram employed in the description of the image acquisition unit shown in FIG. 3 .
FIG. 5 is a diagram employed to explain the first region division unit shown in FIG. 3 .
FIG. 6 is a diagram employed to explain the second region division unit shown in FIG. 3 .
FIG. 7 is an internal configuration diagram of the second region division unit shown in FIG. 3 .
8 is an example of a class hierarchy employed in an embodiment of the present invention.
9 to 11 are diagrams employed in the description of the second region division unit shown in FIG. 3 .
12 to 14 are diagrams employed to explain the coloring part shown in FIG. 3 .
15 is a flowchart illustrating a method for automatically coloring a sketch image according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

1 is a system configuration diagram in which an automatic sketch image coloring apparatus according to an embodiment of the present invention is employed.

The system of FIG. 1 may include a user terminal 100 , a network 200 , and a server 300 .

The user terminal 100 may transmit a sketch image to be colored according to a user's manipulation to the server 300 through the network 200 . For example, the sketch image may include an outline indicating one or more objects (eg, an appearance character, a surrounding object, etc.). The sketch image may be referred to as a line drawing image.

The user terminal 100 may generate or store a sketch image to be colored. For example, the sketch image may be an image in which coloring is not completed. That is, the sketch image may include an uncolored portion or may be an image that is not colored at all. In the embodiment of the present invention, a sketch image to be colored means an image that is not colored at all.

When the sketch image to be colored is transmitted to the server 300 , the server 300 performs a first region dividing technique (eg, an edge-based image region dividing technique) and a second region dividing technique (eg, semantic) with respect to the sketch image. Each region is divided by image region division technique).

The server 300 may generate one or more colored unit areas by merging the divided areas by the first area division technique and the divided areas by the second area division technique, and a corresponding class may be mapped to each colored unit area. have. Here, the class can be said to be a unit having the same meaning as eyes, nose, mouth, hair, arms, legs, and the like.

The server 300 may output, to the user terminal 100 , a result (eg, an image) in which classes are mapped for each colored unit area.

Accordingly, the user terminal 100 may transmit feedback information (eg, class adjustment information) on one or more of the class-mapped coloring unit regions to the server 300 through the network 200 . For example, the class for one colored unit area may be incorrectly mapped to "arm" even though it should be "eye". In this case, after visually checking this, the user may input a class to be corrected with respect to the corresponding colored unit area. Accordingly, the user terminal 100 may transmit feedback information including a class to be corrected for the corresponding colored unit area to the server 300 through the network 200 .

The user terminal 100 is a computing device having an arithmetic processing function, and may be a mobile device or a fixed device. For example, the user terminal 100 may mean a computer, a personal computer (PC), a smart phone, a navigation system, a laptop computer, a tablet computer, a wearable device, or a tablet, but embodiments of the present invention are not limited thereto.

Although only one user terminal 100 is illustrated in FIG. 1 , it is better to understand that there are actually a plurality of user terminals 100 .

The user terminal 100 may communicate with the server 300 through the network 200 . The user terminal 100 may exchange image data with the server 300 according to a wireless communication network or a wired communication network.

For example, the wireless communication type network 200 includes a wireless LAN (WLAN), a wireless broadband (Wibro), a wideband CDMA (WCDMA), an IEEE 802.16, and a long term evolution (LTE). )), LTE-A (Long Term Evolution-Advanced), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, adjacent magnetic field Communication (Near Field Communication (NFC)), Ultrasound Communication (Ultra Sound Communication (USC)), Visible Light Communication (VLC), Wi-Fi, etc. may include, but are limited thereto not.

On the other hand, the network 200 of the wired communication method is wired LAN (Local Area Network), wired WAN (Wide Area Network), power line communication (Power Line Communication (PLC)), USB communication, Ethernet (Ethernet), serial communication (serial) communication), and optical/coaxial cable communication, but is not limited thereto.

The server 300 may be an apparatus for automatically coloring sketch images according to an embodiment of the present invention. Of course, if necessary, the automatic sketch image coloring apparatus according to the embodiment of the present invention may be included in the server 300 .

The server 300 receives (obtains) a sketch image to be colored from the user terminal 100 . Of course, the server 300 may receive a color request for the sketch image together with the sketch image. If a sketch image is received from each user terminal 100 , the server 300 may receive identification information for each user terminal 100 together.

The server 300 may perform region division on the sketch image using the first region division technique (eg, an edge-based image region division technique).

Also, the server 300 may divide the sketch image into a region using a second region division technique (eg, a semantic image region division technique). In the case of region division by the second region division technique, a class may be mapped for each pixel of the sketch image, and the sketch image may be divided into one or more divided regions based on the class mapped for each pixel. In addition, the server 300 may automatically verify and adjust the class mapped for each pixel based on the inclusion relationship between the preset classes. Here, the verification and adjustment means verifying whether the mapped class is normal and automatically adjusting it to the normal class if it is abnormal.

The server 300 may generate one or more colored unit areas by merging the divided areas by the first area division technique and the divided areas by the second area division technique, and a corresponding class may be mapped to each colored unit area. have.

The server 300 may output, to the user terminal 100 , a result (eg, an image) in which classes are mapped for each colored unit area.

When the server 300 receives user feedback information (eg, class adjustment information) for one or more of the coloring unit areas to which the class is mapped from the user terminal 100 , the server 300 applies the corresponding user feedback information to the corresponding coloring unit area based on the user feedback information. class can be adjusted.

The server 300 may color the sketch image to be colored by using a color corresponding to a class mapped to each of the coloring unit areas.

For example, for coloring, the server 300 may store color information allocated for each class in advance. For example, the server 300 may store the class palette in advance. In the case of the class palette, a color for each class, such as class 1 is flesh color, class 2 is red, etc., may be predetermined. Alternatively, the color for each class of the class palette may be determined based on color information for each class in the previous picture. In the case of a webtoon, it may be composed of a plurality of episodes (which may be referred to as "a story"), and each episode may include a plurality of cuts. Accordingly, for example, when it is desired to color the sketch image of episode 2, color information for each class used for coloring the sketch image of episode 1 can be used as it is.

As another example, for coloring, the server 300 may store a plurality of reference coloring images (or may also be referred to as "sample coloring images"). The reference colored image may be an image constituting the webtoon. For example, the reference colored image may include outlines and colors representing objects (eg, characters, surrounding objects, etc.) in the webtoon. That is, the reference colored image may include an image divided by a first region division technique (eg, an edge-based image region division technique) and color information of each division region.

To this end, the server 300 may extract (or collect) a colored image that may be a reference colored image from the webtoon. For example, the server 300 may extract (or collect) a reference colored image from a published webtoon by using the address of a webpage on a web on which the webtoon is published, but is not limited thereto.

According to embodiments, the server 300 may store the reference colored image and order information of the reference colored image together. In this case, the order information may indicate an order in each webtoon of the reference colored image. For example, the order information may mean information about an episode of a webtoon including a reference colored image or a cut including a reference colored image.

The server 300 may receive a sketch image to be colored from the user terminal 100 , and may determine a reference colored image to be used for coloring the sketch image from among previously stored colored images. In this case, the reference colored image may be an image including an object similar to an object in the sketch image.

The server 300 may color the sketch image to be colored by using the determined reference colored image. In this case, the server 300 compares any one coloring unit area (A) in the sketch image to be colored and the coloring unit area (a, b, c, ?) in the reference coloring image with each other. Then, when the server 300 finds a coloring unit area (eg, a) similar to the coloring unit area A in the sketch image to be colored, the color of the area (ie, the coloring unit area a) is retrieved from the sketch image. The color unit area (A) of In this way, color information corresponding to other coloring unit areas of the sketch image to be colored may also be extracted and colored.

The server 300 may provide a colored result to the user terminal 100 when the coloring of the sketch image, which is the coloring target, is completed. In this case, the server 300 may receive feedback from the user terminal 100 information on the part to be color corrected. Accordingly, the server 300 may perform color correction on the corresponding color unit area based on user feedback information (ie, information on a part to be color corrected).

The above-described server 300 may be a computing device capable of communicating with the user terminal 100 and having an arithmetic processing function.

FIG. 2 is an internal configuration diagram of the server 300 shown in FIG. 1 .

The server 300 may include a communication unit 10 , a storage unit 20 , and a control unit 30 .

The communication unit 10 may exchange data with the user terminal 100 . The communication unit 10 may exchange image data with the user terminal 100 according to a wireless communication method or a wired communication method.

The storage unit 20 may store data necessary for the operation of the server 300 . According to embodiments, the storage unit 20 may store a sketch image to be colored, color information assigned to each class, a reference colored image for reference when coloring the sketch image, and the like.

Also, the storage unit 20 may store a program composed of instructions for performing a series of operations performed by the server 300 .

The storage unit 20 may include a non-volatile memory device or a volatile memory device. According to embodiments, the storage unit 20 may be configured to be included in the control unit 30 , but is not limited thereto.

The controller 30 may control overall operations of the server 300 .

For example, the controller 30 may divide the sketch image to be colored using the first region division technique. The controller 30 may divide the sketch image to be colored using the second region division technique. The controller 30 may generate one or more colored unit areas by merging the divided areas by the first area division technique and the divided areas by the second area division technique, and a corresponding class may be mapped to each colored unit area. have. The controller 30 may color the sketch image with a color corresponding to a class mapped to each of the coloring unit areas. The controller 30 may adjust the class of the corresponding coloring unit area based on user feedback information.

That is, the control unit 30 may refer to a device capable of performing a series of operations or determinations for the operation of the server 300 . For example, the controller 30 may be a central processing unit (CPU), a micro controller unit (MCU), a graphic processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), The present invention is not limited thereto.

3 is an internal configuration diagram of the control unit 30 shown in FIG. 2 , FIG. 4 is a diagram employed in the description of the image acquisition unit 31 shown in FIG. 3 , and FIG. 5 is the first shown in FIG. It is a diagram employed in the description of the region dividing unit 32, FIG. 6 is a diagram employed in the description of the second region dividing unit 33 shown in FIG. 3, and FIG. 7 is the second region division shown in FIG. It is an internal configuration diagram of the installment unit 33, FIG. 8 is an example of a class hierarchy employed in an embodiment of the present invention, and FIGS. 9 to 11 are descriptions of the second region division unit 33 shown in FIG. , and FIGS. 12 to 14 are diagrams employed in the description of the coloring part 36 shown in FIG. 3 .

The controller 30 controls the image acquisition unit 31 , the first region dividing unit 32 , the second region dividing unit 33 , the merging unit 34 , the feedback reflecting unit 35 , and the coloring unit 36 . may include

Each of the components 31 , 32 , 33 , 34 , 35 and 36 of the control unit 30 may be physically separated hardware. At this time, at least two or more of the components 31 , 32 , 33 , 34 , 35 and 36 may be integrally implemented. In addition, each of the components 31 , 32 , 33 , 34 , 35 and 36 may be a functionally distinct module. That is, each of the components 31 , 32 , 33 , 34 , 35 and 36 may mean a functional block performing each function of the control unit 30 , in which case each of the components 31 , 32 , 33 , 34 , 35 and 36 may be implemented by one piece of hardware (eg, a processor).

The image acquisition unit 31 may receive a sketch image (eg, see FIG. 4 ) to be colored from the user terminal 100 through the communication unit 10 .

The first region dividing unit 32 may divide the sketch image by using the first region dividing technique. Here, the regions divided by the first region division technique may be referred to as first unit regions.

For example, the first region segmentation technique may be an edge-based image region segmentation algorithm. Accordingly, the first region dividing unit 32 may divide regions of one or more objects (eg, characters, surrounding objects, etc.) in the sketch image, and divide each object into one or more first regions based on an edge. It can be divided into unit areas.

The first region dividing unit 32 may output an image in which the sketch image is divided into a plurality of first unit regions based on an edge, as shown in FIG. 5 . In FIG. 5 , each of the first unit regions divided by the first region division technique is only displayed in color for better understanding. For example, it shows that when the sketch image of FIG. 4 is divided by the first region division technique, it can be divided into five regions (background, body, face, hair, and eyebrows) as shown in FIG. 5 . In reality, a predetermined color may not be displayed for each first unit area.

The first region dividing unit 32 may allocate a tag to each of the divided first unit regions. For example, the first area dividing unit 32 may assign a predetermined number to each first unit area.

The second region dividing unit 33 may divide the sketch image by using the second region dividing technique. Here, the regions divided by the second region division technique may be referred to as second unit regions.

For example, the second region segmentation technique may be a semantic image region segmentation algorithm. The semantic image region segmentation algorithm classifies each pixel of a sketch image into a specific object and assigns a label. For semantic image region segmentation, it is also possible to design and learn a learning network using CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and FCN (Fully Convolutional Networks).

The second region dividing unit 33 receives the sketch image, which is the coloring object, and encodes and decodes it according to the learned method to obtain a class probability map indicating the probability corresponding to a plurality of classes predetermined for each pixel of the sketch image. have. For example, each pixel may have one or more class probabilities, such as class a probability, class b probability, class c probability, and the like. As described above, a set of one or more class probabilities for each pixel may be referred to as a class probability map. In addition, the second region dividing unit 33 maps (or may be referred to as “allocation”) the class having the highest probability for each pixel, and divides the sketch image into one or more second units based on the class mapped for each pixel. It can be divided into regions.

In this way, the second region dividing unit 33 may output a semantically divided image in which the sketch image is semantically divided into regions (class-mapped) as shown in FIG. 6 . In FIG. 6 , each of the second unit regions divided by the second region division technique is only displayed in color for better understanding. For example, it shows that when the sketch image of FIG. 4 is divided by the second region division technique, it can be divided into a plurality of regions (ie, regions with different colors) as in FIG. 6 . In reality, a predetermined color may not be displayed for each second unit area.

In addition, even if the class with the highest probability is assigned to each pixel, the second region divider 33 verifies whether the assigned class is normal or abnormal, and corrects the abnormal class.

In the embodiment of the present invention, if only the semantic image segmentation algorithm is used, there may be a "segmentation coarse" problem in which the image segmentation of the edge part is not performed properly. Therefore, in the embodiment of the present invention, the first area dividing unit 32 is employed together to solve this problem. That is, this is because the edge-based image region segmentation technique can more finely (or accurately) segment the edges of the sketch image compared to the semantic image segmentation technique.

The second region dividing unit 33 may include a learning unit 41 and a dividing unit 42 as shown in FIG. 7 .

The learning unit 41 may use a predetermined training data set for learning. Here, the training data set may be a collection of sketch images obtained from other webtoons and sketch images of previous episodes (episodes). The training dataset can provide pixel-level labels for various semantic classes.

Of course, colored images can be included in the training data set if desired. That is, if a colored image is included in the training data set and the training is performed after tagging, it is possible to know where each pixel is included (eg, eyes, nose, mouth, etc.).

The division unit 42 performs semantic image division on the currently input sketch image according to the learning result of the learning unit 41 .

For example, the division unit 42 obtains an input feature map f by encoding an input image according to a method previously learned by the learning unit 41, and uses the obtained input feature map f in a previously learned method. By decoding according to , an input class probability map (P) is obtained. Here, the input image is an image to be a semantic image segmentation target and a subsequent coloring target, and may be a sketch image received from the image acquisition unit 31 . And, the division unit 42 analyzes the probability value for each class of each pixel in the input class probability map P, and maps, for example, the class having the maximum value to the class of the corresponding pixel (or may be referred to as "allocation") )do. Accordingly, the segmentation unit 42 may output a semantic segmented image (ie, an image in which classes are mapped for each second unit region) with respect to the input image.

In particular, the segmentation unit 42 may not only perform semantic image segmentation, but may also perform verification and automatic adjustment of classes mapped to pixels within each semantically segmented second unit region. To this end, the dividing unit 42 may include an adjusting unit 42a.

The adjuster 42a may correct the class having an error by verifying the class mapped to each pixel of each second unit area.

For example, the adjustment unit 42a may perform class verification and automatic adjustment according to the class hierarchy diagram as in FIG. 8 . The class hierarchy can be seen as representing the containment relationship between classes. A class for each layer of the class hierarchy may be predetermined.

In the case of FIG. 8, the image includes a person and a background, a person includes a head and a torso, a head includes a face and hair, a torso includes an upper body and a lower body, and the face includes eyes and nose , may mean that the right arm and the left arm belong to the upper body, and the right leg and the left leg belong to the lower body. The adjustment unit 42a may verify and automatically adjust the class mapped for each pixel based on such an inclusion relationship.

In FIG. 8 , the person and the background may be a first class hierarchy, the head and torso may be a second class hierarchy, the face and hair, upper and lower body may be a third class hierarchy, and eyes and nose With the right arm and the left arm and the right leg and the left leg can be the 4th class tier.

Here, the first class layer may be the highest layer, and the fourth class layer may be the lowest layer. The second class hierarchy is higher than the third class hierarchy, but may be lower than the first class hierarchy.

Also, classes in the same hierarchy are mutually exclusive. Exclusive means that, for example, if it is assumed that two classes (A, B) belong to the class hierarchy, pixels in the area belong to class B if they do not belong to class A. In other words, exclusive means that if it belongs to one of the classes of the same hierarchy, it does not belong to the other class.

The adjustment unit 42a may verify the correctness of the class mapping by gradually moving from the upper class hierarchy to the lower class hierarchy. Since the class mapping process for each class hierarchy is independent of each other, the calculation of the class mapping probability in each class hierarchy must be performed for the entire class belonging to each class hierarchy. That is, in the embodiment of the present invention, for example, even if the class mapping result for the head/body belongs to the head, in the subclass mapping, the face, hair (hereinafter, belonging to the head class), upper body, lower body (hereinafter, belonging to the body class) belonging) must perform class mapping for all of them. This is to prevent overdetection and erroneous detection.

Thereafter, during class mapping in the lower class hierarchy, the mediator 42a excludes from the class mapping a candidate that does not belong to a previously determined upper class hierarchy among the lower class candidates. In other words, the adjustment unit 42a obtains the probability that each of the classes in the current class hierarchy is mapped for each pixel, and the class having the highest probability among the classes included in the class having the highest probability in the upper class hierarchy. is determined as the class of the current class hierarchy. In other words, the class having the highest probability among classes in the current class hierarchy may not be included in the class having the highest probability in the previously determined upper class hierarchy. If so, the class having the highest probability is not determined as the class of the current class hierarchy, but the class having the highest probability among the remaining classes except for the class having the highest probability is determined as the class of the current class hierarchy. In this way, the class (that is, the meaning) mapped for each pixel of the second unit area can be more accurately adjusted.

For example, if a class called "head" is mapped with respect to a pixel of one second unit area, class mapping to a lower class hierarchy is performed. That is, the class probability values of the face, hair, upper body, and lower body of the third class hierarchy that are lower than the hierarchy to which the class "head" belongs (that is, the second class hierarchy) are calculated. Here, it is assumed that the class probability value of the face is 0.2, the class probability value of the hair is assumed to be 0.1, the class probability value of the upper body is assumed to be 0.4, and the class probability value of the lower body is assumed to be 0.3. In a state in which such class probability values are calculated, if the adjuster 42a is not present, the class of the pixel of the corresponding second unit area will be incorrectly mapped to the upper body.

However, since the adjusting unit 42a is employed in the embodiment of the present invention, in the above example, the adjusting unit 42a can correctly adjust the class of pixels in the second unit area to the class of the face. That is, even if the class probability values are calculated as described above, the adjusting unit 42a may adjust the class of the pixel in the second unit region to a face having a higher class probability value among faces and hair belonging to the head. Accordingly, in the case of the above example, the adjustment unit 42a can correctly adjust what is incorrectly mapped to the class of the upper body as the class of the face.

In the above description, it has been described that the class for each pixel of the second unit region is verified and automatically adjusted by the second region division technique. However, if necessary, the class for each second unit region may be verified and automatically adjusted.

Due to the addition of the function of the adjustment unit 42a, the accuracy of the predicted and generated model for coloring becomes higher.

The use of the adjustment unit 42a will be described once again as follows.

If an image such as the sketch image 50 of FIG. 9 is received, the second region divider 33 may output a semantic segmented image such as the lower image 52 as a result of model prediction.

However, since class verification and automatic adjustment are performed by the adjuster 42a, the second region segmentation unit 33 may finally output the semantic segmented image 56 with class adjustment for each pixel as shown in FIG. 10 . . That is, in FIG. 9 , since the pixels of the second unit area 54 belong to the body and are incorrectly class-mapped to the face, the pixels of the second unit area 54 are determined by class verification in the adjustment unit 42a. As in 10, it is coordinated with a class called torso. In other words, since only an upper body and a lower body may belong to the body, the pixels of the corresponding second unit area 54 are adjusted to the class of the body. In FIGS. 9 and 10 , each of the second unit regions divided by the second region division technique is only displayed in color for better understanding. For example, if the sketch image 50 of FIG. 9 is divided by the second region dividing technique, it may be divided into a plurality of second unit regions (that is, regions with different colors) as shown in FIG. 10 . shows In reality, a predetermined color may not be displayed for each second unit area.

Of course, the class verification and automatic adjustment as described above may be performed while the user directly visually confirms the adjustment. However, in the embodiment of the present invention, the adjustment unit 42a as described above is added to perform its functions, thereby eliminating the trouble of the user having to adjust the class after visually verifying the class. For this reason, there is an effect of providing convenience in use.

And, since the learning unit 41 of the second region dividing unit 33 learns by including the information of the sketch image of the previous episode in the learning data set, the second region dividing unit 33 is, for example, in the case of two episodes. can output an improved semantic segmented image 62 without errors with respect to the sketch image 60 as shown in FIG. 11 compared to the first episode. In FIG. 11 , each of the second unit regions divided by the second region division technique is only displayed in color for better understanding. For example, when the sketch image 60 in the upper part of FIG. 11 is divided by the second region segmentation technique, a plurality of regions (ie, regions with different colors) as in the semantic segmentation image 62 of FIG. 11 . ) can be divided into In reality, a predetermined color may not be displayed for each second unit area.

In FIG. 3 , the merging unit 34 merges the first unit regions by the first region divider 32 and the second unit regions for the second region divider 33 to form one or more colored units. A region may be created, but a corresponding class may be mapped (determined) to each colored unit region.

The merging unit 34 is based on the first unit region by the first region dividing unit 32 . This is because the first region dividing unit 32 divides the image of the edge portion more accurately than the second region dividing unit 33 .

When the first unit regions by the first region dividing unit 32 and the second unit regions of the second region dividing unit 33 are merged, each of the first unit regions by the first region dividing method It may contain one or more pixels. At this time, each pixel is mapped to a respective class (which may have been previously adjusted by the adjusting unit 42a).

Accordingly, the merging unit 34 may determine the representative class as the class of the first unit area by checking the classes of all pixels in the corresponding area for each first unit area by the first area division technique. For example, if the class (ie, meaning) of each pixel in the specific first unit area is mostly "eye", the specific first unit area may be determined as "eye".

In other words, when regions of one sketch image are divided using the first region division technique and the second region technique, there is a slight difference between them. Accordingly, in the embodiment of the present invention, since the first unit area is more accurately divided along the edge than the second unit area, it can be seen that the first unit areas are determined as the colored unit areas. In addition, it can be seen that a class corresponding to each color unit region is mapped based on the class for each pixel by the second region division technique.

Accordingly, it can be seen that the merging unit 34 generates one or more colored unit areas, and outputs an image having information about the type of each colored unit area while a tag is attached to each colored unit area. Here, the information on the type of region may be referred to as information indicating what meaning (ie, class) for each colored unit region.

Meanwhile, the merging unit 34 may transmit a result of mapping classes for each colored unit area to the user terminal 100 through the communication unit 10 and the network 200 . This is to obtain feedback information (eg, class adjustment information) from the user side. Of course, class mapping for each colored unit area would have been normally performed due to class verification and automatic adjustment by the adjusting unit 42a, but it can be regarded as an operation necessary to correct an unexpected error.

The user will not transmit feedback information if there is no error by visually checking the result of class mapping for each colored unit area displayed on his/her terminal 100 .

If a class for one or more coloring unit areas is incorrectly mapped, the user manipulates his/her terminal 100 to transmit feedback information (eg, class adjustment information) including a class to be corrected for the corresponding coloring unit area to the network ( 200) through the server 300.

In the above-described embodiment of the present invention, before merging (overlapping) the division results in the first region dividing unit 32 and the second region dividing unit 33, the divided region (ie, the second It has been described that merging is performed after class adjustment of pixels within the unit area). However, different from this, it is considered that after merging (overlapping) the division results in the first region dividing unit 32 and the second region dividing unit 33, class adjustment for each division based on the class hierarchy is performed. may be However, before merging, the merging unit 34 may output a more accurate result by adjusting the erroneous result of the second region dividing unit 33 in advance and then merging.

The feedback reflecting unit 35 may receive user feedback information (eg, class adjustment information for adjusting a class mapped to one or more colored unit areas) from the user terminal 100 through the communication unit 10 .

In this case, the class adjustment for the color unit region may be directly performed by the feedback reflecting unit 35 or the class adjustment may be performed by the merging unit 34 .

For example, the feedback reflecting unit 35 receives a result of mapping classes for each colored unit area from the merging unit 34 , and when receiving user feedback information from the user terminal 100 , You can adjust the class directly.

As another example, the feedback reflecting unit 35 provides the merging unit 34 with user feedback information (eg, class adjustment information for adjusting a class mapped to one or more colored unit areas) from the user terminal 100 , The merging unit 34 may be configured to adjust the class of the color unit area to be adjusted.

The coloring unit 36 receives the result from the merging unit 34 or the feedback reflecting unit 35 (that is, a sketch image on which class mapping and adjustment for each coloring unit area has been completed), and corresponds to the class mapped for each coloring unit area. You can color the sketch image using the color you want.

There may be two types of coloring methods in the coloring unit 36 .

The first method uses color information pre-allocated for each class. In this method, color information for each class should be stored in advance in the storage unit 20 or the coloring unit 36 . For example, a class palette in which color information for each class is predetermined, such as flesh color for class 1, red color for class 2, black color for class 3, etc. must be stored in advance. And, the color information for each class may be determined based on the color information for each class in the previous picture.

When the first method is adopted as described above, the coloring unit 36 may color the sketch image based on color information pre-allocated for each class.

The second method is to use a reference colored image. In this method, a plurality of reference colored images (or may also be referred to as “sample colored images”) should be previously stored in the storage unit 20 or the coloring unit 36 . Here, the reference colored image may include outlines and colors indicating objects (eg, characters, surrounding objects, etc.) in the webtoon. For example, the reference colored image may be divided into a plurality of divided regions, and may include color information of each divided region.

In this case, the divided region of the reference colored image may be region-divided by the first region-divisioning technique, by the second region-segmentation technique, or by a combination of the first region dividing technique and the second region dividing technique. have.

In the case of adopting the second method, the coloring unit 36 may extract color information corresponding to each coloring unit area of the sketch image by comparing the sketch image (ie, the uncolored image) with the reference coloring image. Then, the coloring unit 36 may color the coloring unit area in the sketch image based on the extracted color information.

Here, the above-described comparison will be described in more detail as follows.

First, based on a deep learning algorithm, a pixel in a divided region (eg, a colored unit region) of a sketch image and a pixel in a divided region of a reference colored image are converted into vectors, respectively. The transformed vector may have a position of a corresponding pixel, information about neighboring pixels, and the like. In addition, the transformed vector may have a value indicating the shape, shape, outline, etc. of the corresponding divided region. For example, the deep learning algorithm may be an algorithm generated through learning using pre-prepared learning images as a training data set, so that vectors having similar characteristics (shape, shape, appearance, etc.) have similar directions to each other.

Using the vectors generated as described above, a degree of similarity between a vector associated with each pixel of the divided region of the sketch image and a vector associated with each pixel of the divided region of the reference colored image is determined. That is, the degree of similarity between the pixels of the divided region of the sketch image and the pixels of the divided region of the reference colored image is determined. For example, as the directions of two vectors compared to each other are similar, the degree of similarity between two pixels corresponding thereto may be calculated as a high value.

The coloring unit 36 determines (calculates) the similarity between the pixels of the divided area of the sketch image and the pixels of the divided area of the reference colored image, and sets the color of the pixels of the divided area of the sketch image, the similar pixels among the pixels of the reference colored image. It can be colored with For example, the coloring unit 36 may color each divided area of the sketch image with a color of the corresponding divided area of the reference colored image similar to (ie, most similar to) each divided area of the sketch image. In this case, for example, the coloring unit 36 may color the color of each pixel of the divided area of the sketch image with one color representing the color of pixels of the divided area of the reference colored image corresponding to the pixels. have. For example, the one representative color may be a color generated through a weighted sum of the pixels and colors of pixels of a division region of the reference colored image corresponding to the pixels. In this case, a weight for each pixel used for the weighted sum may be determined based on the above-described similarity. As described above, each coloring unit area of a sketch image to be colored may be collectively colored with a color (eg, a color that is weighted) determined to be most suitable for each coloring unit area.

Meanwhile, the coloring unit 36 may perform a color correction operation during coloring.

That is, the color unit 36 learns the difference between the color result (that is, the color result before color correction) that is the output of the color model and the correct image to calculate the reliability value for each color unit area (or per pixel) of the sketch image. can In order to calculate the reliability value for each region of the sketch image, the coloring unit 36 may use a predetermined reliability calculation deep learning model. Here, the reliability calculation deep learning model may receive a sketch image and a reference colored image as inputs, and may output a reliability value for each colored unit area of the sketch image.

In other words, until color correction is performed, the coloring unit 36 may color the sketch image by region by calculating the similarity of each region between the reference colored image and the sketch image using the coloring deep learning model. In this case, the coloring deep learning model only informs the calculation results about the similarity and coloring for each area it has calculated, but does not tell how accurately the results are reliable. Accordingly, it can be seen that the coloring unit 36 separately creates and learns a deep learning model capable of calculating the reliability of those calculated by the coloring deep learning model (ie, a reliability calculation deep learning model).

As a result, the coloring unit 36 determines (calculates) the region reliability of each of the coloring unit regions of the sketch image, and the color of the coloring unit region having the determined region reliability lower than a predetermined reference value is applied to other coloring unit regions. It can be colored (i.e. color corrected) with the corresponding color. In some embodiments, the coloring unit 36 may be configured to have the same shape as the correction target region among coloring unit regions of the sketch image with respect to a coloring unit region (hereinafter, a correction target region) having a region reliability lower than a predetermined reference value. Coloring (ie, color correction) may be performed with a color corresponding to a similar color unit area.

For example, in FIG. 13 , it is assumed that the first eye area 70 is an area having area reliability lower than the reference value (eg, about 55%), and the second eye area 72 is an area having area reliability higher than the reference value. do.

In the case of the above example, the color corresponding to the eye is correctly colored in the second eye region 72 . However, the first eye region 70 is actually an eye but may be incorrectly colored with a color that does not correspond to the eye.

In order to remove such a case of erroneous coloring, the coloring unit 36 corrects the color of the corresponding area (eg, 70) based on the color of another area (eg, 72) corresponding to the corresponding area (eg, 70). do. Here, the correspondence may mean that the corresponding region and the shape are the same or similar. On the other hand, the correspondence may mean that classes mapped to the corresponding region are the same. Accordingly, as shown in FIG. 14 , the color of the first eye area 70 may be corrected to be the same as the color of the second eye area 72 .

If necessary, only the reliability calculation deep learning model may not be used when calculating the reliability for each colored unit area of the sketch image. For example, by adding a random noise value or giving randomness (using dropout, etc.) to the learned parameters of the colored deep learning model, the difference between several obtained values is calculated. If the difference in the values of the results is small and stable even with various noises (that is, if the standard deviation of several result values is small), the reliability is judged to be high. is large), the reliability is judged to be low. A reliability value can be calculated by adding the reliability calculation result calculated based on such a judgment criterion and the result value of the above-described reliability calculation deep learning model.

Of course, in addition to the automatic color correction based on the reliability described above, the coloring unit 36 may be configured according to a color correction command for a specific coloring unit area from the user terminal 100 (that is, it may be referred to as user feedback information). Color correction may be performed on the specific colored unit area.

15 is a flowchart illustrating a method for automatically coloring a sketch image according to an embodiment of the present invention.

First, the server 300 receives a sketch image to be colored (ie, an image that is not colored) from the user terminal 100 ( S10 ).

Accordingly, the server 300 divides the sketch image into a region using the first region division technique (eg, an edge-based image region division technique) (S12). Regions divided by the first region division technique are referred to as first unit regions.

At the same time, the server 300 performs semantic region division on the sketch image using the second region division technique (eg, a semantic image region division technique) (S14). Regions divided by the second region division technique are referred to as second unit regions. At the same time, the server 300 maps the class for each pixel in each divided area (ie, the second unit area) by the second area division technique. Then, the server 300 verifies the class mapped for each pixel of each second unit area and corrects the erroneous class.

Then, the server 300 merges the divided areas (ie, the first unit area) by the first area division technique and the divided areas (ie, the second unit area) by the second area division technique to color one or more A unit area is generated, and a corresponding class is mapped to each colored unit area (S16). That is, the server 300 uses each of the first unit regions by the first region division technique as a coloring unit region, and selects a class corresponding to each color unit region based on the class for each pixel by the second region division technique. It can be seen as mapping.

Thereafter, the server 300 transmits to the user terminal 100 the result of mapping classes for each colored unit area in step S16 (S18). This is to obtain user feedback information (eg, class adjustment information) from the user side.

However, if there is no error in the result of class mapping for each color unit area, the server 300 will not receive user feedback information (eg, class adjustment information).

However, if a class for one or more colored unit areas is incorrectly mapped, the server 300 receives user feedback information (eg, class adjustment information) from the user terminal 100 .

Accordingly, the server 300 adjusts the class for the corresponding colored unit area based on the user feedback information (eg, class adjustment information) from the user terminal 100 ( S18 ).

Then, the server 300 colors the sketch image to be colored based on the color corresponding to the class finally determined for each coloring unit area (S20). Of course, the server 300 may perform color correction on an incorrectly colored area after being colored. In this case, the color correction is performed according to a color correction command for a specific color unit area from the user terminal 100 (that is, it may be referred to as user feedback information), in addition to automatic color correction based on reliability. color correction can be performed.

In addition, the method for automatically coloring webtoons of the present invention described above can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium is distributed in a computer system connected through a network, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention pertains.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10: communication unit 20: storage unit
30: control unit 31: image acquisition unit
32: first region dividing unit 33: second region dividing unit
34: merging unit 35: feedback reflecting unit
36: coloring part 41: learning part
42: division part 42a: adjustment part
100: user terminal 200: network
300 : server

Claims (20)

a first region dividing unit that divides a sketch image to be colored into first unit regions by using a first region dividing technique;
The sketch image is divided into class-mapped second unit regions using a second region division technique, a class is mapped for each pixel of the sketch image, and the sketch image is generated based on the class mapped for each pixel. a second region dividing unit dividing into two unit regions;
a merging unit generating one or more color unit regions to which classes are mapped by using the first unit regions and the second unit regions; and
a coloring unit for coloring the sketch image using a color corresponding to a class mapped to each of the coloring unit areas;
The coloring part,
After coloring the sketch image, region reliability for each of the pre-generated coloring unit regions is determined, and a color colored in a coloring unit region having a determined region reliability lower than a reference value is corrected, from among the coloring unit regions of the sketch image. Correcting with a colored color in another colored unit area of the same or similar shape;
Sketch image automatic coloring device. The method of claim 1,
The second region dividing unit,
Verification and automatic adjustment of the class mapped for each pixel based on the inclusion relationship between classes,
Sketch image automatic coloring device. 3. The method of claim 2,
The merging unit,
generating one or more colored unit areas by mapping each pixel class for which the verification and automatic adjustment has been completed to each pixel of the first unit areas;
Sketch image automatic coloring device. 3. The method of claim 2,
The second region dividing unit,
Verifies the correctness of class mapping in a way that gradually moves from the upper class hierarchy to the lower class hierarchy,
Sketch image automatic coloring device. 5. The method of claim 4,
The second region dividing unit,
For each pixel, the probability of mapping each of the classes in the current class hierarchy is obtained, and the class having the highest probability among the classes included in the class having the highest probability in the upper class hierarchy is determined as the class of the current class hierarchy doing,
Sketch image automatic coloring device. 6. The method of claim 5,
The second region dividing unit,
If the class having the highest probability among the classes in the current class hierarchy is not included in the class having the highest probability in the previously determined upper class hierarchy, the class having the highest probability is determined as the class of the current class hierarchy Determining the class having the highest probability among the remaining classes except for the class having the highest probability as the class of the current class hierarchy,
Sketch image automatic coloring device. The method of claim 1,
The coloring part,
A plurality of divided regions and a reference colored image including color information of each of the plurality of divided regions are compared with the sketch image to extract color information corresponding to each colored unit region of the sketch image, and based on the extracted color information coloring the sketch image,
Sketch image automatic coloring device. delete delete The method of claim 1,
The coloring part,
performing color correction on the color unit area according to a color correction command from the user terminal;
Sketch image automatic coloring device. The method of claim 1,
and a feedback reflecting unit that receives user feedback information for adjusting the class mapped for each colored unit area from the user terminal and transmits it to the merging unit.
Sketch image automatic coloring device. The method of claim 1,
a feedback reflecting unit that directly adjusts a class for a color unit area to be adjusted based on user feedback information from a user terminal among the classes mapped for each color unit area;
Sketch image automatic coloring device. a first step of dividing a sketch image to be colored into first unit regions using a first region division technique;
The sketch image is divided into class-mapped second unit regions using a second region division technique, a class is mapped for each pixel of the sketch image, and the sketch image is generated based on the class mapped for each pixel. a second step of dividing into two unit regions;
a third step of generating one or more color unit areas to which classes are mapped by using the first unit areas and the second unit areas; and
a fourth step of coloring the sketch image using a color corresponding to a class mapped to each of the coloring unit areas;
The fourth step is
After coloring the sketch image, region reliability for each of the pre-generated coloring unit regions is determined, and a color colored in a coloring unit region having a determined region reliability lower than a reference value is corrected, from among the coloring unit regions of the sketch image. Correcting with a colored color in another colored unit area of the same or similar shape;
How to automatically color sketch images. 14. The method of claim 13,
The second step is
Including the step of verifying and automatically adjusting the class mapped for each pixel based on the inclusion relationship between the classes,
How to automatically color sketch images. 15. The method of claim 14,
The third step is
generating one or more colored unit areas by mapping each pixel class for which the verification and automatic adjustment has been completed to each pixel of the first unit areas;
How to automatically color sketch images. 15. The method of claim 14,
The second step is
For each pixel, the probability of mapping each of the classes in the current class hierarchy is obtained, and the class having the highest probability among the classes included in the class having the highest probability in the upper class hierarchy is determined as the class of the current class hierarchy doing,
How to automatically color sketch images. 17. The method of claim 16,
The second step is
If the class having the highest probability among the classes in the current class hierarchy is not included in the class having the highest probability in the previously determined upper class hierarchy, the class having the highest probability is determined as the class of the current class hierarchy Determining the class having the highest probability among the remaining classes except for the class having the highest probability as the class of the current class hierarchy,
How to automatically color sketch images. delete delete A computer-readable storage medium storing a program including instructions for performing the method for automatically coloring a sketch image according to claim 13.

Images