WO2020155907A1

WO2020155907A1 - Method and apparatus for generating cartoon style conversion model

Info

Publication number: WO2020155907A1
Application number: PCT/CN2019/126379
Authority: WO
Inventors: 李华夏
Original assignee: 北京字节跳动网络技术有限公司
Priority date: 2019-01-30
Filing date: 2019-12-18
Publication date: 2020-08-06
Also published as: CN109816589B; CN109816589A

Abstract

Disclosed are a method and apparatus for generating a cartoon style conversion model. A particular embodiment of the method comprises: acquiring a training sample set; acquiring a pre-established generative adversarial network; and utilizing a machine learning method to take sample images comprised in training samples in the training sample set as inputs of a generation network, take sample cartoon style images corresponding to the input sample images as expected outputs of the generation network and take cartoon style images actually output by the generation network and the sample cartoon style images corresponding to the input sample images as inputs of a judgment network, training the generation network and the judgment network, and determining the trained generation network to be a cartoon style conversion model. According to the embodiment, the problems such as image edge sawteeth and image contour deformation, generated relative to original images, of the generated cartoon style images can be reduced, such that the display effect of the generated cartoon style images is improved.

Description

Method and device for generating comic style conversion model

Cross references to related applications

This application is filed based on a Chinese patent application with an application number of 201910090075.6 and an application date of January 30, 2019, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

Technical field

The embodiments of the present disclosure relate to the field of computer technology, and in particular to a method and apparatus for generating a comic style conversion model.

Background technique

At present, some image processing software can process images and convert them into images of other styles. For example, change the color and shape of certain areas in the image. These software usually include models for converting images into other styles of images. The training process of the model usually involves inputting an image, comparing the image with its corresponding images of other styles, and optimizing the parameters of the model according to the difference between the two images.

Summary of the invention

The embodiments of the present disclosure propose a method and apparatus for generating a comic style conversion model, and a method and apparatus for generating a comic style image.

In a first aspect, an embodiment of the present disclosure provides a method for generating a comic style conversion model. The method includes: obtaining a training sample set, wherein the training sample includes a preset sample image and a sample corresponding to the sample image Comic style images; obtain a pre-established generative confrontation network, where the generative confrontation network includes a generation network and a discrimination network. The generation network is used to generate comic style images from the input sample images, and the discrimination network is used to distinguish the comic style output of the generation network The image and the sample comic style image corresponding to the sample image of the input generation network; using machine learning methods, the sample image included in the training sample in the training sample set is used as the input of the generation network, and the sample comic style image corresponding to the input sample image As the expected output of the generation network, and the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, the generation network and the discrimination network are trained, and the trained generation network Determined as a manga style conversion model.

In some embodiments, the discriminant network is a multi-scale discriminator, which is used to discriminate the input image and output at least two discriminant results, wherein, for the discriminant result of the at least two discriminant results output, the discriminant result corresponds to The block image included in the image input to the discrimination network and the sample comic style block image included in the sample comic style image are used to determine whether the block image and the corresponding sample comic style block image match.

In some embodiments, training the generation network and the discrimination network includes: using a preset loss function to determine the generation loss value that is used to characterize the difference between the comic style image output by the generation network and the corresponding sample comic style image, and Determine the discriminant loss value corresponding to the discriminant network, which is used to characterize the difference between the comic style image actually output by the generation network of the input discriminant network and the sample comic style image; based on the determined generation loss value and discriminant loss value, the generation network is judged The network is trained.

In some embodiments, the generated loss value is determined by any one of the following loss functions: L1 norm loss function, L2 norm loss function.

In a second aspect, an embodiment of the present disclosure provides a method for generating a comic style image, the method includes: acquiring a target image; inputting the target image into a pre-trained comic style conversion model to generate a comic style image corresponding to the target image And output, wherein the comic style conversion model is generated according to the method described in any one of the embodiments of the first aspect.

In some embodiments, the target image is an image frame extracted from the target video.

In a third aspect, an embodiment of the present disclosure provides an apparatus for generating a comic style conversion model. The apparatus includes: a first acquiring unit configured to acquire a training sample set, wherein the training sample includes a preset sample image , And a sample comic style image corresponding to the sample image; the second acquisition unit is configured to acquire a pre-established generative confrontation network, wherein the generative confrontation network includes a generation network and a discriminant network, and the generation network is used to use the input sample image Generate comic style images, and the discrimination network is used to distinguish between the comic style images output by the generation network and the sample comic style images corresponding to the sample images of the input generation network; the training unit is configured to use machine learning methods to combine the training samples in the training sample set The included sample image is used as the input of the generation network, the sample comic style image corresponding to the input sample image is used as the expected output of the generation network, and the comic style image actually output by the generation network and the sample comic style corresponding to the input sample image The image is used as the input of the discriminant network, the generation network and the discriminant network are trained, and the trained generation network is determined as the comic style conversion model.

In some embodiments, the training unit includes: a determination module configured to use a preset loss function to determine a generation loss value that is used to characterize the difference between the comic style image output by the generation network and the corresponding sample comic style image, and determine The discriminant loss value corresponding to the discriminant network and used to characterize the difference between the comic style image actually output by the generation network of the input discriminant network and the sample comic style image; the training module is configured to be based on the determined generation loss value and discriminant loss value, Train the generation network and the discriminant network.

In a fourth aspect, an embodiment of the present disclosure provides an apparatus for generating a comic style image. The apparatus includes: an image acquisition unit configured to acquire a target image; an image generation unit configured to input the target image into pre-training The comic style conversion model generates and outputs a comic style image corresponding to the target image, where the comic style conversion model is generated according to the method described in any one of the embodiments of the first aspect.

In a fifth aspect, the embodiments of the present disclosure provide an electronic device that includes: one or more processors; a storage device on which one or more programs are stored; when one or more programs are Multiple processors execute, so that one or more processors implement the method described in any one of the first aspect or the second aspect.

In a sixth aspect, the embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of the first aspect or the second aspect is implemented .

The method and device for generating a comic style conversion model provided by the embodiments of the present disclosure are obtained by acquiring a training sample set and a pre-established generating confrontation network, and using a machine learning method, the sample image included in the training sample in the training sample set is taken as The input of the generation network, the sample comic style image corresponding to the input sample image is used as the expected output of the generation network, and the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as the judgment network Input, train the generation network and the discriminant network, and determine the trained generation network as the comic style conversion model. In this way, on the basis of training the generative network using the supervised training method, the training of the generative adversarial network including the generative network and the discriminant network is added, which helps to reduce the generation of the model trained by the supervised training method. Over-fitting problem, improve the generalization ability of the model, and improve the detailed processing ability of image style conversion. Using the cartoon style conversion model obtained by training can reduce the image edge jagged and image contour generated by the generated comic style image relative to the original image Deformation and other issues, thereby improving the display effect of the generated comic style image.

Description of the drawings

By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present disclosure will become more apparent:

Fig. 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure can be applied;

2 is a flowchart of an embodiment of a method for generating a comic style conversion model according to an embodiment of the present disclosure;

3 is a schematic diagram of an application scene of the method for generating a comic style conversion model according to an embodiment of the present disclosure;

4 is a flowchart of an embodiment of a method for generating a comic style image according to an embodiment of the present disclosure;

Fig. 5 is a schematic structural diagram of an embodiment of an apparatus for generating a comic style conversion model according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of an embodiment of an apparatus for generating comic style images according to an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of an electronic device suitable for implementing embodiments of the present disclosure.

detailed description

The present disclosure will be further described in detail below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described here are only used to explain the relevant disclosure, but not to limit the disclosure. In addition, it should be noted that, for ease of description, only the parts related to the relevant disclosure are shown in the drawings.

It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other if there is no conflict. Hereinafter, the present disclosure will be described in detail with reference to the drawings and in conjunction with embodiments.

FIG. 1 shows a method for generating a comic style conversion model or an apparatus for generating a comic style conversion model to which an embodiment of the present disclosure can be applied, and a method for generating a comic style image or a method for generating a comic style image An exemplary system architecture 100 of the device.

As shown in FIG. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the

terminal devices

101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

The user can use the

terminal devices

101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, and so on. Various communication client applications, such as image processing applications, web browser applications, instant messaging tools, and social platform software, may be installed on the

terminal devices

101, 102, and 103.

The

terminal devices

101, 102, and 103 may be hardware or software. When the

terminal devices

101, 102, 103 are hardware, they may be various electronic devices. When the

terminal devices

101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. It can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or as a single software or software module. No specific restrictions are made here.

The server 105 may be a server that provides various services, for example, a back-end server that processes a collection of training samples uploaded by the

terminal devices

101, 102, and 103. The background server can use the acquired training sample set to train the generative adversarial network to obtain a comic style conversion model. In addition, the background server can also use the comic style conversion model to process the input image to obtain the comic style image and output.

It should be noted that the method for generating a comic style conversion model provided by the embodiments of the present disclosure can be executed by the server 105, or can be executed by the

terminal devices

101, 102, 103, and accordingly, used to generate the comic style conversion model The device of can be set in the server 105 or in the

terminal equipment

101, 102, 103. In addition, the method for generating comic style images provided by the embodiments of the present disclosure can be executed by the server 105, and can also be executed by the

terminal devices

101, 102, 103. Accordingly, the device for generating comic style images can be set in The server 105 may also be provided in the

terminal devices

101, 102, 103.

It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules for providing distributed services), or as a single software or software module. No specific restrictions are made here.

It should be understood that the numbers of terminal devices, networks, and servers in FIG. 1 are merely illustrative. According to implementation needs, there can be any number of terminal devices, networks and servers. In the case that the training sample set required for training the model or the target image to be converted into a comic-style image does not need to be obtained remotely, the foregoing system architecture may not include a network, but only a server or terminal device.

Continuing to refer to FIG. 2, a process 200 of an embodiment of the method for generating a comic style conversion model according to the present disclosure is shown. The method for generating a comic style conversion model includes the following steps:

Step 201: Obtain a training sample set.

In this embodiment, the executor of the method for generating a comic style conversion model (for example, the server or terminal device shown in FIG. 1) can obtain the training sample collection remotely or locally through a wired connection or a wireless connection. The training samples include preset sample images and sample comic style images corresponding to the sample images. Generally, the sample image may be an image obtained by photographing a real person, object, scene, etc. The sample comic style image may be an image obtained by performing various processings on its corresponding sample image (for example, changing colors, shapes, and adding special effects to certain areas in the image), and the resulting image usually has a comic style. For example, the character image, background image, etc. in the sample image are processed such as changing colors and adding lines to make them have a comic style. The corresponding relationship between the sample image and the sample comic style image is established in advance. For example, the technician may process the sample image in advance for each sample image in the plurality of sample images, so as to determine the image obtained after the processing as the sample comic style image corresponding to the image.

Step 202: Obtain a pre-established generative confrontation network.

In this embodiment, the above-mentioned execution subject may obtain a pre-established generational confrontation network locally or remotely. Among them, the generative confrontation network includes the generative network and the discriminant network. The generation network is used to generate a comic style image using the input sample image, and the discrimination network is used to distinguish the comic style image output by the generation network and the sample comic style image corresponding to the sample image input to the generation network. The above-mentioned generative confrontation network may be a generative confrontation network of various structures. For example, the generative confrontation network may be a deep convolutional generation confrontation network (Deep Convolutional Generative Adversarial Network, DCGAN). It should be understood that the above-mentioned generative confrontation network may be an untrained generative confrontation network after initializing parameters, or a trained generative confrontation network.

It should be noted that the generating network may be a convolutional neural network used for image processing (for example, a convolutional neural network with various structures including a convolution layer, a pooling layer, a depooling layer, and a deconvolution layer). The above-mentioned discriminant network may be a convolutional neural network (for example, a convolutional neural network of various structures including a fully connected layer, where the above-mentioned fully connected layer can implement a classification function). In addition, the discriminant network may also be other models for implementing classification functions, such as Support Vector Machine (SVM). Here, for each image input to the discrimination network, if the discrimination network determines that the image is the image output by the generation network, it can output the label 1 (or 0) corresponding to the image; if it is determined that the image is not output by the generation network , You can output the label 0 (or 1) corresponding to the image. It should be noted that the discrimination network can also output other preset information, which is not limited to the values 1 and 0.

Step 203: Using a machine learning method, the sample images included in the training samples in the training sample set are used as the input of the generation network, the sample comic style images corresponding to the input sample images are used as the expected output of the generation network, and the actual network is generated. The output comic style image and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, the generation network and the discrimination network are trained, and the trained generation network is determined as the comic style conversion model.

In this embodiment, the above-mentioned execution subject can use a machine learning method to use the sample images included in the training samples in the training sample set as the input of the generation network, and use the sample comic style images corresponding to the input sample images as the expectations of the generation network. The output, and the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, the generation network and the discrimination network are trained, and the trained generation network is determined as a comic style conversion model.

Specifically, the above-mentioned executive body may first fix the parameters of any one of the generation network and the discrimination network (may be called the first network), and optimize the network with no fixed parameters (may be called the second network); then fix it The parameters of the second network are optimized for the first network. Continuously carry out the above iterations, so that the judgment network cannot distinguish whether the input image is generated by the generation network. At this time, the comic style image generated by the generation network is close to the sample comic style image, and the discrimination network cannot accurately distinguish the comic style image generated by the generation network from the sample comic style image (that is, the discrimination accuracy rate is 50%). The generation network is determined to be a comic style conversion model. Generally, the above-mentioned executive body can use the existing back propagation algorithm and gradient descent algorithm to train the generation network and the discriminant network. The parameters of the generative network and the discriminant network after each training will be adjusted, and the generative network and the discriminant network obtained after each parameter adjustment are used as the generative confrontation network used in the next training. In the training process, the loss value can be determined by using the loss function, and the generation network and the discriminant network are iteratively trained according to the loss value to minimize the loss value determined during each iteration.

It should be noted that when training the generation network, the method adopted in this embodiment is to use the sample images included in the training samples in the training sample set as the input of the generation network, and use the sample comic style image corresponding to the input sample image As the expected output, the generation network is trained, and the actual output can be obtained for the sample image input for each training. Among them, the actual output is the comic style image actually output by the generated network. Then, the above-mentioned execution body can adopt the back propagation algorithm and the gradient descent algorithm to adjust the parameters of the generation network based on the actual output and the expected output, and use the generation network after each adjustment of the parameters as the generation network used for the next training.

In some optional implementation manners of this embodiment, the above-mentioned execution subject may train the generation network and the discrimination network according to the following steps:

Firstly, using a preset loss function, determine the generation loss value used to characterize the difference between the comic style image output by the generation network and the corresponding sample comic style image (ie, the sample comic style image corresponding to the input sample image), and determine It is used to characterize the discriminant loss value of the difference between the comic style image actually output by the generated network and the sample comic style image input to the discriminant network.

Generally, the generated loss value can be the loss value determined according to the regression loss function. The regression loss function is generally expressed as L(y, y′), and the loss value obtained by it is used to represent the true value (that is, the loss value in this embodiment) The degree of inconsistency between the sample image) y and the predicted value (that is, the comic style image output by the generation network in this embodiment) y′. During training, the generation loss value is minimized. In addition, a loss function for two classification (for example, a cross-entropy loss function) can be used to determine the discriminative loss value.

Optionally, the generated loss value may be determined by any one of the following loss functions: L1 norm loss function, L2 norm loss function. Among them, the L1 norm loss function and the L2 norm loss function are the existing pixel-level loss functions, that is, the pixel is the basic unit to determine the difference between the pixels included in the two images, which can improve the use of the generated loss value characterization The accuracy of the differences between the images.

Then, based on the determined generation loss value and discrimination loss value, the generation network and the discrimination network are trained. Specifically, the preset weights corresponding to the generated loss value and the discriminated loss value can be used to perform a weighted summation on the determined loss values to obtain the total loss value. During training, constantly adjust the parameters of the generation network and the discrimination network, so that the total loss value gradually decreases, when the total loss value meets the preset condition (for example, less than or equal to the preset loss value threshold, or the total loss value no longer decreases) When it is determined that the training of the confrontation network is completed.

In some optional implementations of this embodiment, the discrimination network may be a multi-scale discriminator, which is used to discriminate the input image and output at least two discrimination results. Wherein, for the discrimination result in the at least two discrimination results output, the discrimination result corresponds to the block image included in the comic style image input to the discrimination network and the sample comic style block image included in the sample comic style image, the discrimination result Used to determine whether the block image matches the corresponding sample comic style block image.

The comic style image input to the judgment network is the comic style image output by the generated network, and the divided image included is obtained by dividing it. The sample comic style segmented image included in the sample comic style image is obtained by dividing the sample comic style image. Generally, each discrimination result corresponds to a block image and a sample comic style block image, and the corresponding relationship between the discrimination result and the block image and the sample comic block image is preset. As an example, the aforementioned at least two discrimination results may be a matrix with N rows and N columns, where N is a preset positive integer. Each element in the matrix is the result of discrimination. Each discrimination result corresponds to a block image and a sample comic block image. That is, the discrimination network separately divides the input comic style image and the sample comic style image to obtain N×N block images and N×N comic style block images. There is a one-to-one correspondence between the block image and the comic style block image. The discriminant network sequentially discriminates the corresponding block image and the comic style block image, and obtains the corresponding discrimination result. Generally, the discrimination result can be the number 0 or 1. 0 can indicate that the corresponding block image does not match the comic style block image, and 1 can indicate that the corresponding block image matches the comic style block image. When the number of discrimination results that characterize the match is greater than or equal to the preset number, it is determined that the discrimination network cannot accurately distinguish the comic style image generated by the generation network and the sample comic style image (that is, the discrimination accuracy rate is 50%).

In practice, the above-mentioned multi-scale discriminator may be a convolutional neural network with a PatchGAN structure. PatchGAN can effectively identify high-frequency components in an image and improve the accuracy of discriminating details in the image. This helps to improve the accuracy of judging the comic style image and the sample comic style image generated by the generation network.

Continue to refer to FIG. 3, which is a schematic diagram of an application scenario of the method for generating a comic style conversion model according to the present embodiment. In the application scenario of FIG. 3, the electronic device 301 first obtains the training sample set 302 locally. Wherein, each training sample in the training sample set 302 includes a preset sample image and a sample comic style image corresponding to the sample image. The sample comic image is the image obtained by the technicians who change the color, shape, and add special effects of the sample image in advance. Then, the electronic device 301 obtains the pre-established generation confrontation network 303 locally. The generation confrontation network 303 includes a generation network 3031 and a discrimination network 3032. The generation network 3031 is used to generate a comic style image using input sample images, and the discrimination network 3032 is used to determine whether the image input to the discrimination network is an image output by the generation network. Then, using a machine learning method, the sample images included in the training samples in the training sample set 302 are used as the input of the generation network, the sample comic style image corresponding to the input sample image is used as the expected output of the generation network, and the generation network The actual output comic style image and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, and the generation network and the discrimination network are trained. Among them, when training the generation network, the comic style image output by the generation network is compared with the corresponding sample comic style image, and the judgment result of the comic style image output by the generation network and the corresponding sample comic style image through the discrimination network , Adjust the parameters of the generated network. When the discrimination network cannot accurately distinguish the comic style image output by the generation network and the sample comic style image (that is, the discrimination accuracy rate is 50%), the generation network at this time is determined as the comic style conversion model 304.

The method provided by the above-mentioned embodiments of the present disclosure obtains a training sample set and a pre-established generative confrontation network, and uses a machine learning method to use the sample image included in the training sample in the training sample set as the input of the generative network, which will be compared with the input The sample comic style image corresponding to the sample image is used as the expected output of the generation network, and the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, and the generation network and the discrimination network are processed Training, the generated network after training is determined as a comic style conversion model. In this way, on the basis of training the generative network using the supervised training method, the training of the generative adversarial network including the generative network and the discriminant network is added, which helps to reduce the generation of the model trained by the supervised training method. Over-fitting problem, improve the generalization ability of the model, and improve the detail processing ability of image style conversion. Using the manga style conversion model obtained by training can reduce the jagged edges of the generated manga style image relative to the original image and reduce the image Contour distortion and other issues, thereby improving the display effect of the generated comic style image.

With further reference to FIG. 4, it shows a process 400 of an embodiment of a method for generating a comic style image according to the present disclosure. The process 400 of the method for generating comic style images includes the following steps:

Step 401: Obtain a target image.

In this embodiment, the execution subject of the method for generating comic style images (for example, the server or terminal device shown in FIG. 1) may obtain the target image remotely or locally through a wired connection or a wireless connection. Among them, the target image is an image to be used to generate a comic style image. For example, the target image may be an image obtained by photographing a target object by a camera included in the above-mentioned execution subject or a camera included in an electronic device communicatively connected with the above-mentioned execution subject, and the target object may be a person, thing, or scene within the shooting range of the camera. Wait.

In some optional implementation manners of this embodiment, the foregoing target image is an image frame extracted from the target video. Wherein, the target video may be a video whose image frames are to be converted into comic style images. For example, the target video may be a video currently played on the execution subject, or a video currently being shot by a camera included in the execution subject or an electronic device communicatively connected with the execution subject. The above-mentioned execution master may extract an image frame from the target image as the target image. As an example, the target image may be an image frame included in the target video and currently displayed on the display screen included in the execution subject.

Step 402: Input the target image into a pre-trained comic style conversion model to generate and output a comic style image corresponding to the target image.

In this embodiment, the above-mentioned execution subject may input the target image into a pre-trained comic style conversion model to generate and output a comic style image corresponding to the target image. The comic style conversion model is generated according to the method described in the embodiment corresponding to FIG. 2 above.

The above-mentioned execution subject can output the generated comic style image in various ways. For example, the generated comic style image may be displayed on a display screen included with the execution subject, or the generated comic style image may be sent to other electronic devices communicatively connected with the execution subject.

The method provided in the above-mentioned embodiment of the present disclosure obtains a target image, inputs the target image into a comic style conversion model trained in advance according to the method described in the above-mentioned embodiment corresponding to FIG. 2 to generate a comic-style image corresponding to the target image and output it, using this The comic style conversion model can reduce the over-fitting problem of generating comic style images, and improve the effect of comic style conversion on the details in the image, and can reduce the image edge jagged of the generated comic style image relative to the original image. Image contour distortion and other issues, thereby improving the display effect of the generated comic style image.

With further reference to FIG. 5, as an implementation of the method shown in FIG. 2, the present disclosure provides an embodiment of a device for generating a comic style conversion model, which is similar to the method embodiment shown in FIG. Correspondingly, the device can be specifically applied to various electronic devices.

As shown in FIG. 5, the apparatus 500 for generating a comic style conversion model of this embodiment includes: a first obtaining unit 501 configured to obtain a set of training samples, where the training samples include preset sample images and A sample comic style image corresponding to the image; the second acquisition unit 502 is configured to acquire a pre-established generative confrontation network, wherein the generation of the confrontation network includes a generation network and a discrimination network, and the generation network is used to generate a comic style using the input sample image The discriminant network is used to distinguish the comic style image output by the generation network and the sample comic style image corresponding to the sample image of the input generation network; the training unit 503 is configured to use a machine learning method to include the training samples in the training sample set The sample image is used as the input of the generation network, the sample comic style image corresponding to the input sample image is used as the expected output of the generation network, and the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as The input of the discriminant network is trained, the generative network and the discriminant network are trained, and the trained generative network is determined as the comic style conversion model.

In this embodiment, the first obtaining unit 501 may obtain the training sample set remotely or locally through a wired connection or a wireless connection. The training samples include preset sample images and sample comic style images corresponding to the sample images. Generally, the sample image may be an image obtained by photographing a real person, object, scene, etc. The sample comic style image may be an image obtained by performing various processings on its corresponding sample image (for example, changing colors, shapes, and adding special effects to certain areas in the image), and the resulting image usually has a comic style. For example, the character image, background image, etc. in the sample image are processed such as changing colors and adding lines to make them have a comic style. The corresponding relationship between the sample image and the sample comic style image is established in advance. For example, the technician may process the sample image in advance for each sample image in the plurality of sample images, so as to determine the image obtained after the processing as the sample comic style image corresponding to the image.

In this embodiment, the second acquiring unit 502 may acquire a pre-established generative confrontation network locally or remotely. Among them, the generative confrontation network includes the generative network and the discriminant network. The generation network is used to generate a comic style image using the input sample image, and the discrimination network is used to distinguish the comic style image output by the generation network and the sample comic style image corresponding to the sample image input to the generation network. The above-mentioned generative confrontation network may be a generative confrontation network of various structures. For example, the generative confrontation network may be a deep convolutional generation confrontation network (Deep Convolutional Generative Adversarial Network, DCGAN). It should be understood that the above-mentioned generative confrontation network may be an untrained generative confrontation network after initializing parameters, or a trained generative confrontation network.

In this embodiment, the training unit 503 can use a machine learning method to use the sample images included in the training samples in the training sample set as the input to the generation network, and use the sample comic style images corresponding to the input sample images as the expectations of the generation network. Output, and use the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image as the input of the discrimination network, train the generation network and the discrimination network, and determine the generated network after training as the comic style conversion model.

Specifically, the above-mentioned training unit 503 may first fix the parameters of any one of the generation network and the discrimination network (may be called the first network), and optimize the network without fixed parameters (may be called the second network); and then Fix the parameters of the second network and optimize the first network. Continuously carry out the above iterations, so that the judgment network cannot distinguish whether the input image is generated by the generation network. At this time, the comic style image generated by the generation network is close to the sample comic style image, and the discrimination network cannot accurately distinguish the comic style image generated by the generation network from the sample comic style image (that is, the discrimination accuracy rate is 50%). The generation network is determined to be a comic style conversion model. Generally, the above-mentioned training unit 503 can use the existing back propagation algorithm and gradient descent algorithm to train the generation network and the discrimination network. The parameters of the generative network and the discriminant network after each training will be adjusted, and the generative network and the discriminant network obtained after each parameter adjustment are used as the generative confrontation network used in the next training. In the training process, the loss value can be determined by using the loss function, and the generation network and the discriminant network are iteratively trained according to the loss value to minimize the loss value determined during each iteration.

It should be noted that when training the generation network, the method adopted in this embodiment is to use the sample images included in the training samples in the training sample set as the input of the generation network, and use the sample comic style image corresponding to the input sample image As the expected output, the generation network is trained, and the actual output can be obtained for the sample image input for each training. Among them, the actual output is the comic style image actually output by the generated network. Then, the above-mentioned training unit 503 may adopt a back propagation algorithm and a gradient descent algorithm to adjust the parameters of the generation network based on the actual output and the expected output, and use the generation network after each adjustment of the parameters as the generation network used for the next training.

In some optional implementations of this embodiment, the discriminant network is a multi-scale discriminator, used to discriminate the input image, and output at least two discriminating results, wherein, for the at least two discriminating results output The discrimination result, which corresponds to the block image included in the image input to the discrimination network and the sample comic style block image included in the sample comic style image, is used to determine whether the block image matches the corresponding sample comic style block image.

In some embodiments, the training unit 503 may include: a determining module (not shown in the figure), configured to use a preset loss function to determine the comic style image and the corresponding sample comic style image used to characterize the network output The generation loss value of the difference of the discriminant network, and the discriminant loss value that determines the difference between the comic style image and the sample comic style image corresponding to the discriminant network and used to represent the actual output of the generation network of the input discriminant network; training module (not shown in the figure) , Is configured to train the generation network and the discrimination network based on the determined generation loss value and the discrimination loss value.

The apparatus 500 provided by the above-mentioned embodiment of the present disclosure acquires a training sample set and a pre-established generative confrontation network, and uses a machine learning method to use a sample image included in the training sample in the training sample set as the input of the generative network, and will be compared with the input The sample comic style image corresponding to the sample image of the generation network is used as the expected output of the generation network, and the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, and the generation network and the discrimination network Perform training and determine the trained generation network as a comic style conversion model. In this way, on the basis of training the generative network using the supervised training method, the training of the generative adversarial network including the generative network and the discriminant network is added, which helps to reduce the generation of the model trained by the supervised training method. Over-fitting problem, improve the generalization ability of the model, and improve the detail processing ability of image style conversion. Using the manga style conversion model obtained by training can reduce the jagged edges of the generated manga style image relative to the original image and reduce the image Contour distortion and other issues, thereby improving the display effect of the generated comic style image.

With further reference to FIG. 6, as an implementation of the method shown in FIG. 4, the present disclosure provides an embodiment of a device for generating comic style images, and the device embodiment corresponds to the method embodiment shown in FIG. 4 , The device can be applied to various electronic equipment.

As shown in FIG. 6, the apparatus 600 for generating a comic style image of this embodiment includes: an image acquisition unit 601 configured to acquire a target image; an image generation unit 602 configured to input the target image into a pre-trained comic style The conversion model generates and outputs a comic style image corresponding to the target image. The comic style conversion model is generated according to the method described in the embodiment corresponding to FIG. 2 above.

In this embodiment, the image acquisition unit 601 can acquire the target image remotely or locally through a wired connection or a wireless connection. Among them, the target image is an image to be used to generate a comic style image. For example, the target image may be an image obtained by photographing a target object by a camera included in the foregoing apparatus 600 or an electronic device included in an electronic device communicatively connected with the foregoing apparatus 600, and the target object may be a person, thing, or scene within the shooting range of the camera. Wait.

In this embodiment, the image generating unit 602 may input the target image into a pre-trained comic style conversion model to generate and output a comic style image corresponding to the target image. The comic style conversion model is generated according to the method described in the embodiment corresponding to FIG. 2 above.

The above-mentioned image generating unit 602 can output the generated comic style image in various ways. For example, the generated comic style image may be displayed on a display screen included with the execution subject, or the generated comic style image may be sent to other electronic devices communicatively connected with the execution subject.

In some optional implementation manners of this embodiment, the target image is an image frame extracted from the target video.

The apparatus 600 provided in the above-mentioned embodiment of the present disclosure acquires a target image, inputs the target image into the comic style conversion model trained in advance according to the method described in the above-mentioned embodiment corresponding to FIG. 2, and generates and outputs the comic style image corresponding to the target image. The comic style conversion model can reduce the over-fitting problem of generating comic style images, improve the effect of comic style conversion on details in the image, and reduce the jagged edges of the generated comic style image relative to the original image. Reduce the problem of image contour distortion, thereby improving the display effect of the generated comic style image.

Referring now to FIG. 7, it shows a schematic structural diagram of an electronic device (such as the server or terminal device in FIG. 1) 700 suitable for implementing the embodiments of the present disclosure. Terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals ( For example, mobile terminals such as car navigation terminals and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG. 7 is only an example, and should not bring any limitation to the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 7, the electronic device 700 may include a processing device (such as a central processing unit, a graphics processor, etc.) 701, which may be loaded into a random access device according to a program stored in a read-only memory (ROM) 702 or from a storage device 708. The program in the memory (RAM) 703 executes various appropriate actions and processing. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to the bus 704.

Generally, the following devices can be connected to the I/O interface 705: including input devices 706 such as touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, liquid crystal display (LCD), speakers, vibration An output device 707 such as a device; a storage device 708 such as a magnetic tape and a hard disk; and a communication device 709. The communication device 709 may allow the electronic device 700 to perform wireless or wired communication with other devices to exchange data. Although FIG. 7 shows an electronic device 700 having various devices, it should be understood that it is not required to implement or have all the illustrated devices. It may alternatively be implemented or provided with more or fewer devices. Each block shown in FIG. 7 may represent one device, or may represent multiple devices as needed.

In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable signal medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device is caused to: obtain a training sample set, wherein the training sample includes a preset sample image, and The sample comic style image corresponding to the sample image; obtain a pre-established generative confrontation network, where the generative confrontation network includes a generation network and a discriminant network. The generation network is used to generate a comic style image using the input sample image, and the discriminant network is used to differentiate The comic style image output by the network and the sample comic style image corresponding to the sample image of the input generation network; using machine learning methods, the sample images included in the training samples in the training sample set are used as the input of the generation network, which will correspond to the input sample image The sample comic style image of the generation network is used as the expected output of the generation network, and the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, and the generation network and the discrimination network are trained. The generated network after training is determined to be the comic style conversion model.

In addition, when the above-mentioned one or more programs are executed by the electronic device, the electronic device is caused to: acquire the target image; input the target image into the pre-trained comic style conversion model to generate and output the comic style image corresponding to the target image.

The computer program code used to perform the operations of the embodiments of the present disclosure can be written in one or more programming languages or a combination thereof, the programming languages including object-oriented programming languages such as Java, Smalltalk, C++, It also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to pass Internet connection).

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram can represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more for realizing the specified logic function Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations Or it can be realized by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. The described unit may also be provided in the processor, for example, it may be described as: a processor includes a first acquiring unit, a second acquiring unit, and a training unit. Wherein, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the first obtaining unit can also be described as "a unit for obtaining a training sample set."

The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover the above-mentioned inventive concept without departing from the above-mentioned inventive concept. Other technical solutions formed by any combination of technical features or equivalent features. For example, the above-mentioned features and the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions are replaced with each other to form a technical solution.

Claims

A method for generating comic style conversion models, including:

Acquiring a set of training samples, where the training samples include preset sample images and sample comic style images corresponding to the sample images;

Obtain a pre-established generative confrontation network, wherein the generative confrontation network includes a generation network and a discrimination network, the generation network is used to generate a comic style image using the input sample image, and the discrimination network is used to distinguish the generation network The output comic style image and the sample comic style image corresponding to the sample image input to the generating network;

Using machine learning methods, the sample images included in the training samples in the training sample set are used as the input of the generation network, the sample comic style image corresponding to the input sample images is used as the expected output of the generation network, and the actual output of the generation network The comic style image of and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, the generation network and the discrimination network are trained, and the trained generation network is determined as the comic style conversion model.
The method according to claim 1, wherein the discriminant network is a multi-scale discriminator for discriminating the input image and outputting at least two discriminating results, wherein, for the at least two discriminating results output The discrimination result, which corresponds to the block image included in the image input to the discrimination network and the sample comic style block image included in the sample comic style image, is used to determine whether the block image matches the corresponding sample comic style block image.
The method according to claim 1 or 2, wherein the training of the generating network and the discriminant network comprises:

Using the preset loss function, determine the generation loss value used to characterize the difference between the comic style image output by the generation network and the corresponding sample comic style image, and determine the actual output of the generation network corresponding to the discrimination network and used to characterize the input discrimination network The discriminant loss value of the difference between the comic style image and the sample comic style image;

Based on the determined generation loss value and discrimination loss value, the generation network and the discrimination network are trained.
The method according to claim 3, wherein the generated loss value is determined by any one of the following loss functions: L1 norm loss function and L2 norm loss function.
A method for generating comic style images, including:

Get the target image;

The target image is input into a pre-trained comic style conversion model to generate and output a comic style image corresponding to the target image, wherein the comic style conversion model is generated according to the method of any one of claims 1-4 .
The method according to claim 5, wherein the target image is an image frame extracted from a target video.
A device for generating a comic style conversion model, including:

The first acquiring unit is configured to acquire a training sample set, where the training sample includes a preset sample image and a sample comic style image corresponding to the sample image;

The second acquisition unit is configured to acquire a pre-established generative confrontation network, wherein the generative confrontation network includes a generation network and a discrimination network, and the generation network is used to generate a comic style image using the input sample image, and the discrimination The network is used to distinguish the comic style image output by the generation network and the sample comic style image corresponding to the sample image input to the generation network;

The training unit is configured to use a machine learning method to use the sample images included in the training samples in the training sample set as the input of the generation network, and use the sample comic style image corresponding to the input sample images as the expected output of the generation network, And the comic style image actually output by the generation network and the sample comic style image corresponding to the input sample image are used as the input of the discrimination network, the generation network and the discrimination network are trained, and the trained generation network is determined as the comic style conversion model.
7. The device according to claim 7, wherein the discriminant network is a multi-scale discriminator for discriminating the input image and outputting at least two discriminating results, wherein, for the at least two discriminating results The discrimination result, which corresponds to the block image included in the image input to the discrimination network and the sample comic style block image included in the sample comic style image, is used to determine whether the block image matches the corresponding sample comic style block image.
The device according to claim 7 or 8, wherein the training unit comprises:

The determination module is configured to use a preset loss function to determine the generation loss value that is used to characterize the difference between the comic style image output by the generation network and the corresponding sample comic style image, and to determine the generation loss value corresponding to the discrimination network and used to characterize the input discrimination The generation of the network The judgment loss value of the difference between the comic style image actually output by the network and the sample comic style image;

The training module is configured to train the generation network and the discrimination network based on the determined generation loss value and the discrimination loss value.
The apparatus according to claim 9, wherein the generated loss value is determined by any one of the following loss functions: L1 norm loss function and L2 norm loss function.
A device for generating comic style images, including:

An image acquisition unit configured to acquire a target image;

The image generating unit is configured to input the target image into a pre-trained comic style conversion model to generate and output a comic style image corresponding to the target image, wherein the comic style conversion model is according to claims 1-4 Generated by the method described in one.
The apparatus according to claim 11, wherein the target image is an image frame extracted from the target video.
An electronic device including:

One or more processors;

A storage device on which one or more programs are stored,

When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.
A computer-readable medium with a computer program stored thereon, wherein the program is executed by a processor to implement the method according to any one of claims 1-6.