KR20230163907A - Systen and method for constructing converting model for cartoonizing image into character image, and image converting method using the converting model - Google Patents

Abstract

A method of constructing a conversion model that converts a target image into a cartoonized character image, consisting of a pair of images including a background and a face, respectively, and character images cartoonized with a specific character corresponding to the plurality of images. A method of building a transformation model is provided by creating a training image data set and learning the generated training image data set.

Description

A method and system for constructing a conversion model that cartoonizes an image into a character image, and an image conversion method using the conversion model {SYSTEN AND METHOD FOR CONSTRUCTING CONVERTING MODEL FOR CARTOONIZING IMAGE INTO CHARACTER IMAGE, AND IMAGE CONVERTING METHOD USING THE CONVERTING MODEL}

The present disclosure relates to a method and system for constructing a conversion model for cartooning an image into a character image and a method for converting a target image using the conversion model, where not only the face but also the background of the target image is converted into a cartoonized character image. It is related to a method and system for building a conversion model that allows it to be done.

Interest in services that provide image-type content online, such as comics, cartoons, or webtoon services, is increasing. As interest in such content increases, interest in technology for converting video consisting of images or frames such as photographs into the style or style of the content is also increasing.

Through this technology, real-life images or videos (hereinafter referred to as images) captured by users can be converted into the painting style or style of specific webtoon content. This transformation may be named cartoonization of the image. The converted, that is, cartoonized image may have a color or texture similar to webtoon content. Accordingly, the user can process his or her image into a form similar to the desired webtoon content, and thus can feel that he or she is creating webtoon content or participating in webtoon content.

When converting an image into a cartoonized image, it is required to convert the face included in the image into a character of the content and the background of the image to the painting style or style of the background of the content.

Korean Patent Publication No. 10-2009-0065354 (published on June 21, 2009) relates to a method of converting two-dimensional images into various artistic images. A method of providing images such as oil paintings, pen illustrations, cartoons, double paintings, and template mosaics is disclosed by enabling image conversion using various artistic techniques through image conversion processing.

The information described above is for illustrative purposes only and may include content that does not form part of the prior art and may not include what the prior art would suggest to a person skilled in the art.

One embodiment is a method of building a conversion model for converting a target image into a cartoonized character image, wherein a character is cartoonized as a plurality of images each including a background and a face, and a specific character corresponding to the plurality of images. A method of building a transformation model can be provided by generating a training image data set composed of pairs of images and learning the generated training image data set.

One embodiment uses a transformation model learned through a training image data set consisting of a pair of character images cartoonized with a specific character corresponding to a plurality of images each including a background and a face, and the plurality of images. , a method of converting the target image into a character image with a cartoonized background and face can be provided.

In one aspect, a method of constructing a conversion model for converting a target image into a cartoonized character image, performed by a computer system, includes a plurality of images each including a background and a face, and the plurality of images. A step of generating a training image data set consisting of a pair of character images cartoonized with a corresponding first character, and constructing the transformation model by learning the generated training image data set, wherein the training image data The step of generating the set includes cartoonizing the face of each image of the plurality of images to the first character based on a plurality of learning face images including the face of the first character using a face transformation model. Converting the background of each image into a cartoonized background using a background conversion model, and combining the cartoonized face and the cartoonized background to create a character corresponding to each image. A method for building a transformation model is provided, including generating an image.

The first character is included in content including an image or video, and the step of generating the image data set for learning includes obtaining the learning facial images including the face of the first character from the content. can do.

The content is webtoon content including a plurality of cuts, and the step of acquiring the face images for learning includes extracting a predetermined number of cuts from the content, and from the cuts, at least one including the first character. extracting first facial images including a face of a character, at least one of the first facial images being corrected by performing at least one of alignment, resizing, and resolution change; It may include acquiring face images and clustering the second face images by character, thereby obtaining the learning face images including the face of the first character.

The plurality of images are a first number, the plurality of training face images are a second number smaller than the first number, and the face transformation model is the first number of the plurality of images and the second number. Based on the plurality of facial images for training, the first number of face transformation images corresponding to the first number of plurality of images, including the cartoonized face matching the face of each image. It can be configured to generate

The cartoonized face converted using the face transformation model may be matched with the face in each image in terms of direction, composition, and location of parts included in the face.

The step of converting the face of each image into a cartoonized face as the first character includes defining a latent space of the face conversion model based on the plurality of face images for learning, the latent space It may include generating an inversion code for conversion of each image based on and generating a face conversion image corresponding to each image based on the conversion code.

The first character is included in content including an image or video, the background transformation model is learned based on images that do not contain the character extracted from the content, and the background transformation model is It may be configured to generate a background conversion image by converting the background of the image to a cartoonized background of the content.

The background transformation model may be learned based on real-world images containing only the background.

In the step of generating a character image corresponding to each image, the character image corresponding to each image may be generated by combining the cartoonized face with the background conversion image.

The method of building the transformation model includes inputting the target image including a background and a face to the constructed transformation model, and according to inference based on the transformation model, the face of the target image is the first character. The method may further include generating a converted image in which the face is converted to a cartoonized face, and the background of the target image is converted to the cartoonized background of the content including the first character, and outputting the converted image. there is.

The first character is included in content including an image or video, the content includes a plurality of characters including the first character, and the generating step includes the plurality of images and the plurality of images. Create a learning image data set consisting of a pair of character images cartoonized with each character of the plurality of characters corresponding to, and the conversion model is configured to convert the target image into a character image cartoonized with each character It can be built.

The first character is included in content including an image or video, the content includes a plurality of characters including the first character, and the conversion model converts the target image into a cartoonized character image for each character. and further comprising providing a user interface (UI) for selecting a character to convert the target image from among the plurality of characters, wherein the first character is a character selected through the UI. You can.

The target image is a video including a plurality of frames, and the converted image may be a converted image in which the face and background included in the video are converted into the cartoonized face and the cartoonized background, respectively.

In another aspect, a computer system for building a conversion model for converting a target image into a cartoonized character image, comprising at least one processor implemented to execute instructions readable by the computer system, and the at least one The processor generates a training image data set consisting of a pair of a plurality of images each including a background and a face and cartoonized character images with a first character corresponding to the plurality of images, and the generated training image data set. The transformation model is constructed by learning an image data set, and the at least one processor uses a face transformation model to generate the training image data set, and generates a plurality of learning faces including the face of the first character. Based on the images, the face of each image of the plurality of images is converted into a cartoonized face as the first character, and the background of each image is converted into a cartoonized background using a background conversion model, A computer system is provided that generates a character image corresponding to each image by combining a cartoonized face and the cartoonized background.

In another aspect, in an image conversion method performed by a computer system for converting a target image into a cartoonized character image, a plurality of images each including a background and a face and corresponding to the plurality of images Inputting the target image including a background and a face to a conversion model built by learning a training image data set consisting of a pair of cartoonized character images as a first character, and inference based on the conversion model Accordingly, generating a converted image in which the face of the target image is converted to a cartoonized face of the first character, and the background of the target image is converted to a cartoonized background of content including the first character; and outputting the converted image, wherein the character image corresponding to each image of the plurality of images constituting the training image data set of the conversion model is a face of each image using a face conversion model. The result of converting the first character into a cartoonized face based on a plurality of learning face images including the face of the first character, and the background of each image into a cartoonized background using a background conversion model An image conversion method is provided, which is generated by compositing the result of conversion.

In another aspect, in a method of generating a training image data set for a transformation model for converting a target image into a cartoonized character image, performed by a computer system, using a face transformation model, the first character Based on a plurality of learning face images including the face, converting the face of each image of the plurality of images including the background and the face into a cartoonized face as the first character, using a background conversion model Thus, converting the background of each image into a cartoonized background, combining the cartoonized face and the cartoonized background to generate a character image corresponding to each image, and each image and the cartoonized background. Image data for learning, comprising the step of generating pairs of character images cartoonized with the first character corresponding to the plurality of images and the plurality of images, which are composed of pairs of character images, as a learning image data set. A method for creating sets is provided.

Using a face transformation model, cartoonized face transformation images are created with characters corresponding to multiple real-life images, and cartoonized background transformation images are created using a background transformation model to create face transformation images and background transformation images. Through compositing, a cartoonized character image can be created, and a learning image data set consisting of a pair of real-life images and character images corresponding to the real-life images can be created for the transformation model.

By learning a conversion model with a learning image data set consisting of a pair of real-life images and character images corresponding to the real-life images, through the constructed conversion model, both the face and background of the target image are cartoonized in the style or style of a specific content. A converted image may be created. Therefore, a converted image (character image) with no sense of heterogeneity in the background and face can be created.

Figure 1 shows a method of converting a target image into a character image with a cartoonized background and face using a constructed conversion model, according to one embodiment.
Figure 2 shows a computer system that builds a conversion model for converting a target image into a character image with a cartoonized background and face, according to one embodiment.
Figure 3 is a flowchart showing a method of building a conversion model for converting a target image into a character image with a cartoonized background and face, according to an embodiment.
Figure 4 is a flowchart showing a method of acquiring facial images for learning to build a transformation model, according to an example.
FIG. 5 is a flowchart illustrating a method of generating a face transformation image corresponding to each real-life image using a face transformation model to generate a training image data set for building a transformation model, according to an example.
Figure 6 is a flowchart showing a method of converting a target image into a character image with a cartoonized background and face using a constructed conversion model, according to an embodiment.
7 to 10 show, according to an example, constructing a conversion model for converting a target image into a character image with a cartoonized background and face, and converting the target image into a character image with a cartoonized background and face using the constructed conversion model. This is a flowchart showing how to convert to a character image.
FIG. 11 illustrates a method of generating a character image with a cartoonized face and background corresponding to each real-life image to generate a learning image data set for building a transformation model, according to an example.
12 illustrates a first number of cartoonized faces corresponding to a first number of real-life images, based on a second number of training facial images including faces of characters, using a facial transformation model, according to an example. Indicates how to convert converted images.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 shows a method of converting a target image into a character image with a cartoonized background and face using a constructed conversion model, according to one embodiment.

The computer system 100 may build a conversion model for converting the target image 10 into a character image 50 in which the background and face are cartoonized. When the target image 10 is input to this constructed conversion model, the face 20 of the target image 10 is converted to the face 60 of the character in the conversion image 50 and the background 30 of the target image 10 ) can be converted into the background 70 of the converted image 50.

The target image 10 may be a real image. “Image” can encompass a picture or an image consisting of a frame. In other words, the conversion model can convert an input real image or video into a cartoonized conversion image 50.

The transformation model may be a machine learning or artificial intelligence-based model, for example, a model based on deep learning (DNN) or other artificial neural networks (CNN, ANN, RNN, etc.)

The converted image 50 is a cartoonized image of the target image 10 and may be referred to as a “character image” in the detailed description to be given later. In other words, the converted image 50 may be a cartoonized image of the face 20 and background 30 of the target image 10.

“Cartoonization” of the target image 10 may mean converting the target image 10 into the painting style or style of specific content. Content may include an image or video, and the image or video included in the content may include at least one character. The conversion model of the embodiment creates the conversion image 50 by cartooning the face 20 of the target image 10 as a character of content and cartooning the background 20 of the target image 10 as the background of the content. You can. Accordingly, the converted image 50 may have a color or texture similar to the content. Additionally, the face 60 of the converted image 50 may be expressed as a reflection of the characteristics of a specific character of the content in the characteristics of the face 20 of the target image 10. Characteristics of a character may include, for example, characteristics of the shape, shape, or color of facial areas, such as eye shape. The background 70 of the converted image 50 may be expressed as a reflection of the background characteristics of the content in the characteristics of the background 30 of the target image 10. Background characteristics may include the color or texture of the content.

The content may be cartoon or webtoon content. Alternatively, the content may be a video, such as a cartoon, animation, or movie. A “character” included in content may be an object corresponding to a character in the content. Content may include multiple characters.

The “background” in the embodiment may represent the remaining area in the image excluding the area corresponding to the “face” of the character or person. For example, the background 30 in the target image 10 may be an area excluding the face 20, and the background 70 in the converted image 50 may be an area excluding the face 60.

In the embodiment, through the conversion model, a conversion image 50 is generated in which both the face 20 and the background 30 of the target image 10 are cartooned in the painting style or style of a specific content, thereby creating the background 70 and the background 70. A character image without a sense of heterogeneity can be created on the face 60.

In order to build this conversion model, the computer system 100 uses a learning image data set consisting of a pair of character images cartoonized as a specific character corresponding to a plurality of images each including a background and a face and the plurality of images. can be created, and a transformation model can be built by learning the generated training image data set.

A method of constructing a conversion model by the computer system 100 and a method of converting the target image 10 into a character image 50 using the constructed conversion model will be further described with reference to FIGS. 2 to 12 to be described later. It is explained in detail.

Figure 2 shows a computer system that builds a conversion model for converting a target image into a character image with a cartoonized background and face, according to one embodiment.

The computer system 100 may be a computing device for building a conversion model for converting the above-described target image 10 into a character image 50 in which the background 30 and the face 20 are cartoonized. That is, in order to build a transformation model, the computer system 100 can build a transformation model by creating a training image data set and training the model with the generated training image data set.

Computer system 100 may be configured to include at least one computing device. The computer system 100 may be a server or a part of a server that generates a training image data set and perform a task for learning the generated training image data set, or may be a user terminal or a part of a user terminal. That is, in the embodiments to be described later, the computer system 100 is described as being able to correspond to either a server or a user terminal.

The user terminal is a smart device such as a smart phone, a personal computer (PC), a laptop computer, a laptop computer, a tablet, an Internet of Things device, or a wearable computer ( wearable computer), etc.

When the computer system 100 is a server, the server may be a computing device that is distinct from the user terminal and may be a device that communicates with the user terminal. The computer system 100, which is such a server, can provide a conversion model to the user terminal according to a request received through the user terminal, and therefore, the user terminal uses the conversion model to change the target image 10 to the background ( 30) and the face 20 can be converted into a cartoonized character image 50.

As shown, the computer system 100 may include a memory 130, a processor 120, a communication unit 110, and an input/output interface 140.

The memory 130 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, the ROM and the non-perishable mass recording device may be separated from the memory 130 and included as a separate permanent storage device. Additionally, an operating system and at least one program code may be stored in the memory 130. These software components may be loaded from a computer-readable recording medium separate from the memory 130. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 130 through the communication unit 110 rather than a computer-readable recording medium.

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 120 by the memory 130 or the communication unit 110. For example, the processor 120 may be configured to execute instructions received according to program code loaded into the memory 130.

The communication unit 110 may be a component that allows the computer system 100 to communicate with other devices (such as user terminals or other servers). That is, the communication unit 110 is a hardware module or network, such as an antenna, a data bus, a network interface card, a network interface chip, and a networking interface port, of the computer system 100 that transmits/receives data and/or information to and from other devices. It may be a software module such as a device driver or networking program.

The input/output interface 140 may be a means for interfacing with an input device such as a keyboard or mouse and an output device such as a display or speaker.

The processor 120 is a program or application that can manage the components of the computer system 100, generates a training image data set to build the above-described transformation model, and trains the model with the generated training image data set. can be executed, and operations necessary for executing the program or application and processing data can be processed. The processor 120 may be at least one processor (such as a CPU or GPU) of the computer system 100 or at least one core within the processor.

Additionally, in embodiments, computer system 100 and processor 120 may include more components than those shown. For example, the processor 120 may include components that perform functions for building a transformation model and generating the character image 50 from the target image 10 using the built transformation model. These components of the processor 120 may be part of the processor 120 or may be functions implemented by the processor 120. The components included in the processor 120 may be expressions of different functions performed by the processor 120 according to control instructions according to the code of the operating system or the code of at least one computer program. .

A method of constructing a conversion model performed by the computer system 100 and a method of converting the target image 10 into a character image 50 using the constructed conversion model will be described with reference to FIGS. 3 to 12, which will be described later. This is explained in more detail.

The description of the technical features described above with reference to FIG. 1 can also be applied to FIG. 2 , so overlapping descriptions will be omitted.

In the detailed description to be described later, operations performed by the computer system 100 or the processor 120 or their components may be described as operations performed by the computer system 100 for convenience of explanation.

Figure 3 is a flowchart showing a method of building a conversion model for converting a target image into a character image with a cartoonized background and face, according to an embodiment.

Referring to FIG. 3, a specific method of building a conversion model for converting the target image 10 into a character image 50 in which the background 30 and the face 20 are cartoonized will be described in more detail.

In step 310, the computer system 100 may generate a training image data set for training a transformation model. For example, the computer system 100 may generate a training image data set consisting of a pair of cartoonized character images with a plurality of images each including a background and a face and a first character corresponding to the plurality of images. there is.

Each of the plurality of images may be a real-life image including a background and a face. The plurality of images are randomly collected images and may include at least one person (ie, the person's face).

The first character may be a character included in content for which the target image 10 is to be cartoonized through a transformation model. That is, the first character may be included in content including an image or video, and for example, if the content is webtoon content, it may be one of the characters in the webtoon content.

A training image data set for training a transformation model may include a plurality of images that are real-life images and pairs of character images that are cartoonized as a first character corresponding to the plurality of images. In other words, the training image data set may be composed of a pair of 'each image (each real-life image) - the character image corresponding to each image.' The character image corresponding to each image is a character image matching each image, and the face of each image may be converted into a cartoonized face as the first character and the background of each image may be converted into a cartoonized background. there is.

A specific method of generating a character image matching each image will be described in more detail with reference to steps 312 to 318 to be described later.

In step 320, the computer system 100 may build a transformation model by learning the training image data set generated in step 310. As described above with reference to FIG. 1 , the constructed conversion model can convert the target image 10 input to the conversion model into a character image 50 in which the background 30 and the face 20 are cartoonized.

Below, the method for generating a training image data set will be described in more detail with reference to steps 312 to 318.

When creating a learning image data set for learning the transformation model of the embodiment, a face transformation model and a background transformation model may be used to generate character images corresponding to the plurality of images that are real images.

The face transformation model and the background transformation model may be pre-built on the computer system 100 or may be models pre-built in another computer system accessible to the computer system 100.

At step 312, the computer system 100 uses a facial transformation model to transform the face of each image of the plurality of images into the first character, based on the plurality of training facial images including the face of the first character. You can convert it into a cartoonized face with . That is, the computer system 100 may generate a face conversion image in which the heat curves of each image are converted into a cartoonized face as the first character.

A plurality of face images for learning may be collected as images including the face of the first character of the content. A plurality of face images for learning may be extracted from content. A specific method of acquiring a plurality of facial images for learning will be described in more detail with reference to FIGS. 4 and 7, which will be described later.

A face transformation model may be a model for generating character images corresponding to a larger number of real-life images based on a small number of facial images for learning.

For example, the plurality of images that are real-life images may be a first number, and the plurality of facial images for learning may be a second number smaller than the first number. At this time, the face transformation model is based on a first number of images and a second number of learning face images, and includes a cartoonized face matching the face of each image of the plurality of images. It may be configured to generate a first number of face transformation images corresponding to a plurality of images. In other words, the face transformation model can generate a face transformation image that matches each image of a plurality of images that are real images.

The cartoonized face converted using the face transformation model, that is, the cartoonized face included in the face transformation image, is the face of each image, which is the corresponding real-life image, and the direction, composition, and position of the face of the face. may be matching. In other words, the cartoonized face included in the face conversion image and the face in the corresponding live image face the same direction, have the same overall composition (color, shape, etc.), and have the same location of facial parts (eyes or pupils, nose, etc.) The relative positional relationship of parts such as the mouth and ears), face shape, etc. may be the same.

As such, in the embodiment, by using a face transformation model in generating a training image data set, it is possible to obtain face transformation images corresponding to a larger number of real-life images even by using only a small number of training face images of the first character. You can.

The method of generating a face transformation image using a face transformation model will be described in more detail with reference to FIGS. 5 and 8, which will be described later.

Additionally, when creating a learning image data set for training the transformation model of the embodiment, a background transformation model may be used to cartoonize the background of the plurality of images, which are real-life images, into the background of the content.

In step 314, the computer system 100 may convert the background of each image of the plurality of images, which are real-life images, into a cartoonized background using a background conversion model. That is, the computer system 100 may generate a background conversion image in which the background of each image is cartoonized as the background of content including the first character.

The background transformation model may be learned based on images extracted from content including the first character. The extracted images may be images that do not include characters or other objects.

The background conversion model may be configured to generate a background conversion image by converting the background of each image of a plurality of images, which are real-life images, into a cartoonized background of the content.

The method of generating a background transformation image using a background transformation model will be described in more detail with reference to FIG. 9, which will be described later.

At step 316, the computer system 100 converts the face of each image, which is a real-life image, into a cartoonized face, which is the result of transforming the face of each image using the face transformation model according to step 312, and transforms the background according to step 314. By using the model to synthesize a cartoonized background that is the result of converting the background of each image, which is a real-life image, it is possible to create a character image corresponding to each image.

The computer system 100 may generate a character image by combining the above-described face conversion image and the background conversion image. For example, the computer system 100 may generate character images corresponding to each image, which is a real-life image, by combining the cartoonized face included in the face conversion image with the generated background conversion image.

In step 318, the computer system 100 may generate a training image data set consisting of pairs of the character image generated in step 316 and each image that is a corresponding real-life image.

As a result, a pair of each real-life image and a character image with a cartoonized background and face corresponding to the real-life image can be generated as the final training image data set for the transformation model.

The method of generating a training image data set by combining a face transformation model, a background transformation model, and images will be described in more detail with reference to FIG. 9, which will be described later.

The description of the technical features described above with reference to FIGS. 1 and 2 can also be applied to FIG. 3 , so overlapping descriptions will be omitted.

Figure 4 is a flowchart showing a method of acquiring facial images for learning to build a transformation model, according to an example.

Referring to FIG. 4, the method of obtaining “a plurality of learning face images including the face of the first character” used in generating the learning image data set in the above-described step 310 will be described in more detail. . The plurality of facial images for learning may be images input to a face transformation model to generate face transformation images corresponding to a plurality of images that are real images.

These plurality of face images for learning may be obtained from content including an image or video including the first character. Each of the plurality of facial images for learning may include the face of the first character.

The content may be webtoon content including the first character as a character. Webtoon content may be configured to include at least one episode. Webtoon content or an episode of webtoon content may include multiple cuts.

Below, with reference to steps 410 to 450, a method of obtaining a plurality of facial images for learning from webtoon content when the content is webtoon content will be described in more detail.

At step 410, computer system 100 may extract a predetermined number of cuts from the content.

In step 420, the computer system 100 may extract first facial images including the face of at least one character including the first character from the extracted cuts. Content may include a plurality of characters, and the computer system 100 may extract an image including the face of a character included in each of the extracted cuts as the first face image. The first face image may be an image cropped from a cut to include an area corresponding to the character's face, or may be the cut itself.

In step 430, the computer system 100 may obtain corrected second facial images by processing at least one of the first facial images obtained in step 420. For example, the computer system 100 may obtain corrected second facial images by performing at least one process of alignment, resizing, and resolution change on at least one of the first facial images. .

In step 440, the computer system 100 may classify the second facial images obtained in step 430 by character. For example, the computer system 100 may cluster the second facial images by character.

In step 450, the computer system 100 may obtain training facial images including the face of the first character based on the classification of the second facial images in step 440.

The number of face images for learning may be a relatively small number, for example, 100 or less (or 100 or less or 50 or more). In an embodiment, using the above-described face transformation model, cartoonized face transformation images corresponding to a large number of real-life images can be obtained based on this small number of facial images for training.

Below, with reference to FIG. 7, a method of generating facial images for learning including the face of the first character will be described in more detail.

Facial images for learning may be named 'original data set' of the first character.

Figure 7 shows a method of obtaining the original data set of the first character included in webtoon content from content, according to an example.

In step 710, the computer system 100 may extract images of cuts included in the content by content unit (i.e., work unit), which is webtoon content. For example, the computer system 100 may download webtoon content and extract cut-unit images of the webtoon content using the ToonCutter algorithm. The computer system 100 may extract a predetermined number of cuts, for example, 100 cuts. This may correspond to step 410 described above.

In step 720, the computer system 100 may detect and extract the face of the first character included in the webtoon content from each of the extracted cuts. For example, computer system 100 may use the YoloV4 algorithm, an open source algorithm, to detect the face of a character from a cut. This may correspond to step 420 described above.

In step 730, the computer system 100 may extract facial images corresponding to the face detected in step 720 and perform alignment on them. For example, computer system 100 may use the StyleGAN2 and OpenCV algorithms, which are open source algorithms, for extraction and/or alignment of facial images.

Alignment performed on the extracted face images may correct the direction of the face included in the face image to face the front, and may include resizing the face image. The size of the resized face image may be 256X256.

At step 740, the computer system 100 may perform super-resolution processing on facial images. For example, if the size of the face included in the face image is too small, the computer system 100 can increase the resolution of the face image by using a super-resolution algorithm such as Real-esrgan, which is an open source algorithm. The size of the face image resized according to the super-resolution algorithm may be 1024X1024.

Steps 730 and 740 may correspond to step 430 described above. Steps 730 and 740 may be performed only on facial images that require correction.

In step 750, the computer system 100 may classify the facial images processed according to steps 730 and 740 into images for each character. For example, the computer system 100 is an open source algorithm and can classify facial images into character-specific images using the K-means clustering algorithm. The computer system 100 may acquire facial images of the first character to be used to construct a learning image data set based on the classification results of the facial images as an 'original data set' of the first character. Step 750 may correspond to steps 440 and 450 described above.

The number of facial images for learning of the first character acquired as the 'original data set' may be a relatively small number, for example, 100 or less (or 100 or less or 50 or more).

The description of the technical features described above with reference to FIGS. 1 to 3 can also be applied to FIGS. 4 and 7 , so overlapping descriptions will be omitted.

FIG. 5 is a flowchart illustrating a method of generating a face transformation image corresponding to each real-life image using a face transformation model to generate a training image data set for building a transformation model, according to an example.

Through steps 510 to 530, which will be described later, a face transformation image corresponding to each image of the plurality of images that are real-life images may be generated using a face transformation model.

In step 510, the computer system 100 may define a latent space of a face transformation model based on the plurality of facial images for training acquired in step 450 described above. That is, the computer system 100 can invert the acquired plurality of facial images for learning into the latent space of the face transformation model.

In step 520, the computer system 100 may generate an inversion code for converting each image of the plurality of images that are real images based on the latent space.

In step 530, the computer system 100 may generate a face transformation image corresponding to each image through a face transformation model based on the generated transformation code.

Through the face transformation model of the embodiment, based on a smaller number (e.g., 100 or less) of facial images for learning of the first character, a face corresponding to a larger number (e.g., 10,000 or more) of real-life images Transformed images may be created. Each real image and each face conversion image may have faces that match each other.

In relation to this, below, with reference to FIG. 8, a method for generating a face transformation model and a method for generating a face transformation image corresponding to each real-life image using the face transformation model will be described in more detail.

FIG. 8 shows face transformation images corresponding to real-life images based on the original data set of the first character obtained according to steps 710 to 750 of FIG. 7 using a face transformation model, according to an example. Indicates how to obtain it.

In that the real-life image and the corresponding facial conversion image are paired with each other, the first character, that is, for each character, can be named a "pair data set for learning a conversion model."

In step 810, the computer system 100 selects a plurality of training facial images of the first character obtained by step 450 described above (i.e., the face of the first character obtained by step 750 described above). Based on the images), a face model of the first character may be created. The computer system may generate a facial model of this first character using, for example, the StyleGAN2 algorithm. Specifically, the computer system 100 fine-tunes a data set of a plurality of learning face images of the first character using the Flickr-Faces-HQ (FFHQ) pre-trained StyelGAN2 model and StyleGAN2-ADA model. By learning, a face model of the first character can be created.

In step 820, the computer system 100 may generate a facial transformation model by blending the fine tuning learned model and the FFHQ pre-trained model. Synthesis of these models can be done natively at 32x32 resolution. However, depending on the characteristics of the character, the model may be synthesized at different resolutions. Various blending algorithms can be used to synthesize models.

In step 830, the computer system 100 may invert the plurality of training facial images of the above-described first character into the latent space of the facial transformation model generated according to model synthesis in step 820. . The computer system 100 may generate an inversion code for converting each image of a plurality of images that are real images based on the latent space. The computer system 100 may generate a face transformation image by generating a character face texture corresponding to each image through a face transformation model based on the generated transformation code. The computer system 100 may use, for example, the StyleGAN2 algorithm in generating a character face texture corresponding to each image. This may correspond to steps 510 to 530 described above.

In step 840, the computer system 100 may generate the generated face transformation image and the corresponding real-life image as a paired learning data set.

The plurality of images described above, which are real images, may be randomly collected images. Each of the plurality of images may include a face of a person, and may include the faces of a plurality of people. Images can be collected based on a variety of existing algorithms.

For example, these plurality of images may be real images obtained by decoding a first number (eg, 10000) of random latent codes with an FFHQ pre-trained model. That is, the plurality of images may be 10000 real-life images.

The face transformation model may generate face transformation images corresponding to the plurality of images by combining the random latent codes with a latent code corresponding to the first character and then decoding it. In other words, by performing the decoding through the synthesized model in step 820, 10,000 facial transformation images corresponding to 10,000 real-life images can be generated.

By using this face transformation model, in an embodiment, a large amount of real-life images that can be easily obtained are matched, that is, the direction of the face, the composition of the face, the location of the part(s) included in the face, etc. are matched. , large amounts of facial transformation images can be generated based on the original data set consisting of only a small number of images.

Relatedly, FIG. 12 shows first cartoonized images corresponding to a first number of real-life images based on a second number of training facial images including faces of characters, using a facial transformation model, according to an example. Indicates how to convert a number of face conversion images.

In FIG. 12 , a method of converting a first number of real-life images into a first number of face transformation images using a face transformation model 1200 is schematically shown. In an embodiment, only a second number of training face images that are much smaller than the first number may be required to obtain the first number of facial transformation images.

The face transformation model 1200 encodes each of the live-action images through an encoder, synthesizes the character's latent code according to learning the facial images for learning to a random latent code corresponding to each live-action image, and uses the synthesized code as a decoder. By decoding through , a face transformation image corresponding to each real-life image can be generated.

The face transformation model 1200 may use, for example, the DualStyleGAN algorithm or the JoJoGAN algorithm.

The description of the technical features described above with reference to FIGS. 1 to 4 and FIG. 7 can also be applied to FIGS. 5, 8, and 12, so overlapping descriptions will be omitted.

Figure 9 shows a method of building a transformation model by changing the background of real-life images into a cartoonized background using a background transformation model and generating a final training image data set, according to an example.

At step 910, computer system 100 may collect background data for training a background transformation model. The background data may include images extracted from content including the first character, and the background transformation model may learn this background data. The extracted images may be images that do not include characters or other objects. In other words, each of the extracted images may not contain a face but only natural objects such as trees, mountains, sky, sea, etc. The computer system 100 may build a data set for learning a background transformation model by collecting, for example, 2000 images.

Meanwhile, the computer system 100 may additionally collect real-life images containing only backgrounds, that is, only natural objects, and include the collected real-life images in a data set for learning. For example, the computer system 100 may build a data set for learning a background transformation model by collecting 2000 more real-world images. In other words, the background transformation model of the embodiment may be learned based more on real-world images containing only the background.

The performance of the background transformation model can be improved by including more such real-life images in the data set for learning the background transformation model.

In step 920, the computer system 100 may generate a background transformation model by learning the training data set constructed in step 910. The background transformation model may use, for example, the AnimeGAN algorithm.

In step 940, the background conversion model may convert the background of each image of the plurality of images, which are real-life images, into a cartoonized background of content including the first character. Accordingly, the background transformation model can generate a background transformation image corresponding to each image. Meanwhile, as in step 930, the background conversion model may convert the real-life images collected in step 910 into a cartoonized background of the content including the first character. The result of the transformation in step 930 can be used to build a final training image data set in step 960, which will be described later.

In step 950, the computer system 100 converts the face of each image, which is a photorealistic image, into a cartoonized face using a face transformation model and the background of each image, which is a photorealistic image, using a background conversion model. The cartoonized background that is the result of conversion can be synthesized. That is, the computer system 100 can generate a character image for building a final training image data set by combining the above-described face conversion image and background conversion image. For example, the computer system 100 may generate character images corresponding to each image, which is a real-life image, by combining the cartoonized face included in the face conversion image with the background conversion image. This may correspond to step 316 described above.

The computer system 100 may use, for example, the Ibug face parsing bisenet algorithm to synthesize the face transformation image and the background transformation image.

In step 960, the computer system 100 may finally generate a training image data set consisting of a pair of a character image generated as a result of synthesis in step 950 and a corresponding real-life image. This may correspond to step 318 described above. Meanwhile, the computer system 100 may further use the results of the transformation in step 930 to build a final training image data set. For example, the final training image data set may further include the result of the transformation in step 930.

In relation to this, FIG. 11 shows a method of generating a character image with a cartoonized face and background corresponding to each real-life image in order to generate a training image data set for building a transformation model, according to an example.

As shown, a face in a live-action image may be cartoonized into a first character using the facial transformation model described above. When the face of a live-action image is converted using a face conversion model, the background of the live-action image may be lost, such as being blurred rather than cartoonized, in the converted face conversion image. Therefore, when a transformation model is built using a pair of real-life image - face transformation image as a training image data set for the transformation model, the constructed transformation model can appropriately cartoonize the background 30 of the target image 10. does not exist.

Accordingly, in an embodiment, the background of a real-life image can be cartoonized using a background transformation model. By combining the face conversion image with a background conversion image in which the background is a cartoon as the background of the content, a character image in which both the face and the background of the live-action image are cartoons can be created. In an embodiment, a transformation model is built using this character image-real-life image pair as a training image data set for the transformation model, so that the constructed transformation model includes the face 20 and the background 30 of the target image 10. All of this can be effectively cartoonized.

In step 970, the computer system 100 may generate a transformation model by learning the final training image data set generated in step 960. As described above with reference to FIG. 1 , the generated conversion model can convert the target image 10 input to the conversion model into a character image 50 in which the background 30 and the face 20 are cartoonized. Computer system 100 may use, for example, the DynamicUnet algorithm to generate such a transformation model.

The description of the technical features described above with reference to FIGS. 1 to 5, 7, 8, and 12 can also be applied to FIGS. 9 and 11, so overlapping descriptions will be omitted.

Meanwhile, the conversion model of the embodiment may be implemented to cartoonize the target image 10 with each character of a plurality of characters included in the content. That is, the conversion model of the embodiment is a single model and can support conversion of the target image 10 into the cartoonized conversion image 50 of a plurality of characters included in the content.

This transformation model as a single model can be built in the following way.

For example, in the above-described step 310, the computer system 100 consists of a plurality of images that are real images and a pair of character images cartoonized with each character of a plurality of characters corresponding to the plurality of images. You can create an image data set for learning. The computer system 100 can build a transformation model by learning a training image data set consisting of pairs of 'character images cartoonized for each character - real images'. When constructing a learning image data set for each character, face images clustered for each character in the above-described step 440 (i.e., facial images for learning) can be used. The constructed conversion model can convert the target image into a cartoonized character image for each character.

As a result, the transformation model can be implemented to support cartooning of the target image 10 for each character of the webtoon having various facial abstraction levels.

Figure 6 is a flowchart showing a method of converting a target image into a character image with a cartoonized background and face using a constructed conversion model, according to an embodiment.

Referring to FIG. 6, a method of obtaining the converted image 60 from the target image 10 according to inference using the conversion model will be described in more detail.

The computer system that performs steps 610 to 630 or 1010 to 1030 in the embodiment described later may be a user terminal that generates the converted image 60 from the target image 10 using the constructed conversion model. This user terminal may be a separate device from the computer system 100 that builds the conversion model described above, but for convenience of explanation, it will be described later that the computer system 100 performs steps 610 to 630 or 1010 to 1030. do.

In step 610, the computer system 100 may receive a target image 10 including a background 30 and a face 30 for the constructed transformation model. That is, computer system 100 may input target image 10. The computer system 100 may receive the target image 10 and input it into a transformation model. The target image 10 received by the computer system 100 may be input by the user through a user interface (UI) provided by the computer system 100. The target image 10 may be a video or image selected by the user through a UI, or may be a video or image captured in real time by a camera included in the computer system 100. As described above, the conversion model is constructed by learning a training image data set consisting of a pair of cartoonized character images with a plurality of images each containing a background and a face and a first character corresponding to the plurality of images. It could be. At this time, the character image corresponding to each image of the plurality of images constituting the training image data set of the transformation model is a plurality of learning face images including the face of the first character using the face transformation model. It may be generated by combining the result of converting the first character into a cartoonized face based on the first character and the result of converting the background of each image into a cartoonized background using a background conversion model.

In step 620, the computer system 100 converts the face 20 of the target image 10 into the cartoonized face 60 as the first character, according to reasoning based on the transformation model, and the face 20 of the target image 10 ) of the background 30 may be converted into a cartoonized background 70 of content including the first character to generate a converted image 50.

At step 630, computer system 100 may output converted image 50. The computer system 100 may output the converted image 50 through a display. For example, the computer system 100 may display the converted image 50 instead of the target image 10, or may display the target image 10 and the converted image 50 together so that they can be compared. The computer system 100 may display, in real time, a cartoonized converted image 50 for the target image 10, which is a video or image captured in real time.

Meanwhile, as described above, in the case where the content including the first character includes a plurality of characters, the conversion model converts the target image 10 into a character image 50 cartoonized with each character of the plurality of characters. It may have been built to make this possible.

At this time, as in step 615, the computer system 100 may provide a user interface (UI) for selecting a character to convert the target image 10 from among a plurality of characters in the content. The first character may be a character selected through the UI. The computer system 100 may generate the converted image 50 by cartooning the target image 10 with the selected first character.

In this way, the conversion model of the embodiment can support conversion of the target image 10 into the cartoonized conversion image 50 of a plurality of characters included in the content as a single model. For example, a user may select a character he or she wants from among characters appearing in webtoon content and request conversion of the target image 10 into the painting style or style of the character.

Meanwhile, the target image 10 may be a video including a plurality of frames. At this time, the converted image 50 may be a converted image in which the face 20 and background 30 included in the image are converted into a cartoonized face 60 and a cartoonized background 70.

The conversion model can generate the conversion image 50 by converting the target image 10 on a frame-by-frame basis. If the target image 10 is an image input or captured in real time, the converted image 50 may also be generated in real time.

In relation to this, FIG. 10 explains a method of outputting the converted image 50 from the target image 10 according to inference based on the conversion model.

In step 1010, the target image 10 that is the subject of inference may be input to the constructed transformation model. This may correspond to step 610 described above.

In step 1020, the transformation model may convert the target image 10 into a cartoonized transformation image 50 through inference. This may correspond to step 620 described above.

In step 1020, the computer system 100 may output the generated converted image 50. This may correspond to step 630 described above.

The description of the technical features described above with reference to FIGS. 1 to 5, 7 to 9, 11 and 12 can also be applied to FIGS. 6 and 10, so overlapping descriptions will be omitted.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general-purpose or special-purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (17)

In a method of building a conversion model for converting a target image into a cartoonized character image, performed by a computer system,
Generating a training image data set consisting of a pair of a plurality of images each including a background and a face, and cartoonized character images as a first character corresponding to the plurality of images; and
Building the transformation model by learning the generated training image data set.
Including,
The step of generating the training image data set is,
Using a face conversion model, converting the face of each image of the plurality of images into a cartoonized face of the first character based on a plurality of learning face images including the face of the first character;
Converting the background of each image into a cartoonized background using a background conversion model; and
Creating a character image corresponding to each image by combining the cartoonized face and the cartoonized background.
A method of building a transformation model, including. According to paragraph 1,
The first character is included in content including an image or video,
The step of generating the training image data set is,
Obtaining the learning facial images including the face of the first character from the content
A method of building a transformation model, including. According to paragraph 2,
The content is webtoon content including a plurality of cuts,
The step of acquiring the facial images for learning is,
extracting a predetermined number of cuts from the content;
extracting first facial images including the face of at least one character including the first character from the cuts;
Obtaining corrected second facial images by performing at least one of alignment, resizing, and resolution change on at least one of the first facial images; and
Obtaining the learning facial images including the face of the first character by clustering the second facial images by character.
A method of building a transformation model, including. According to paragraph 1,
The plurality of images is a first number,
The plurality of face images for learning is a second number smaller than the first number,
The face transformation model is,
Based on the first number of the plurality of images and the second number of the plurality of facial images for training, the first number of plurality, including the cartoonized face matching a face in each image A method for building a transformation model, configured to generate the first number of facial transformation images corresponding to images of According to clause 4,
The cartoonized face converted using the face transformation model is,
A method of constructing a transformation model in which the face of each image is matched with the direction, composition, and location of the parts included in the face. According to paragraph 4,
The step of converting the face of each image into a cartoonized face of the first character,
defining a latent space of the face transformation model based on the plurality of facial images for learning;
generating an inversion code for conversion of each image based on the latent space; and
Based on the conversion code, generating a face conversion image corresponding to each image.
A method of building a transformation model, including. According to paragraph 1,
The first character is included in content including an image or video,
The background transformation model is learned based on images that do not contain characters extracted from the content,
The background transformation model is configured to generate a background transformation image by converting the background of each image into a cartoonized background of the content. In clause 7,
A method of building a transformation model, wherein the background transformation model is learned based on real-life images containing only the background. In clause 7,
The step of generating a character image corresponding to each image is,
A method of building a transformation model, wherein a character image corresponding to each image is generated by combining the cartoonized face with the background transformation image. According to paragraph 1,
Inputting the target image including a background and a face into the constructed conversion model;
According to inference based on the conversion model, the face of the target image is converted to a cartoonized face of the first character, and the background of the target image is converted to a cartoonized background of content including the first character. generating a converted image; and
Outputting the converted image
A method of building a transformation model, further comprising: According to paragraph 1,
The first character is included in content including an image or video,
The content includes a plurality of characters including a first character,
The generating step generates a learning image data set consisting of a pair of the plurality of images and cartoonized character images of each character of the plurality of characters corresponding to the plurality of images,
The conversion model is constructed to enable conversion of the target image into a cartoonized character image for each character. According to clause 10,
The first character is included in content including an image or video,
The content includes a plurality of characters including a first character,
The conversion model is constructed to convert the target image into a cartoonized character image for each character,
Providing a user interface (UI) for selecting a character to convert the target image from among the plurality of characters.
It further includes,
A method of building a transformation model, wherein the first character is a character selected through the UI. According to clause 10,
The target image is a video including a plurality of frames,
The converted image is a converted image in which the face and background included in the image are converted into the cartoonized face and the cartoonized background, respectively. A program recorded on a computer-readable recording medium for executing the method of claim 1 on the computer system. In a computer system that builds a conversion model that converts a target image into a cartoonized character image,
At least one processor implemented to execute instructions readable by the computer system
Including,
The at least one processor,
Generate a learning image data set consisting of a pair of cartoonized character images with a plurality of images each including a background and a face and a first character corresponding to the plurality of images,
Constructing the transformation model by learning the generated training image data set,
The at least one processor,
In generating the training image data set,
Using a face conversion model, the face of each image of the plurality of images is converted into a cartoonized face of the first character based on a plurality of learning face images including the face of the first character, and the background is converted. A computer system that uses a model to generate a character image corresponding to each image by converting the background of each image into a cartoonized background and compositing the cartoonized face with the cartoonized background. In an image conversion method performed by a computer system for converting a target image into a cartoonized character image,
For a transformation model built by learning a training image data set consisting of a pair of a plurality of images each containing a background and a face and cartoonized character images with a first character corresponding to the plurality of images, the background and Inputting the target image including a face;
According to inference based on the conversion model, the face of the target image is converted to a cartoonized face of the first character, and the background of the target image is converted to a cartoonized background of content including the first character. generating a converted image; and
Outputting the converted image
Including,
The character image corresponding to each image of the plurality of images constituting the training image data set of the transformation model is,
Using a face conversion model, a result of converting the face of each image into a cartoonized face of the first character based on a plurality of learning face images including the face of the first character;
Results of converting the background of each image into a cartoonized background using a background conversion model
An image conversion method created by compositing. In a method of generating a training image data set for a conversion model for converting a target image into a cartoonized character image, performed by a computer system,
Using a face transformation model, based on a plurality of learning face images including the face of the first character, the face of each image of the plurality of images including the background and the face is cartoonized as the first character. Converting to;
Converting the background of each image into a cartoonized background using a background conversion model;
generating a character image corresponding to each image by combining the cartoonized face and the cartoonized background; and
Generating pairs of character images cartoonized with the plurality of images and the first character corresponding to the plurality of images, consisting of a pair of each image and the character image, as an image data set for learning.
A method of generating an image data set for training, including.

Images