WO2021057463A1

WO2021057463A1 - Image stylization processing method and apparatus, and electronic device and readable medium

Info

Publication number: WO2021057463A1
Application number: PCT/CN2020/113974
Authority: WO
Inventors: 贾靖
Original assignee: 北京字节跳动网络技术有限公司
Priority date: 2019-09-25
Filing date: 2020-09-08
Publication date: 2021-04-01
Also published as: CN110689478B; CN110689478A

Abstract

An image stylization processing method and apparatus, and an electronic device and a readable medium. The method comprises: on the basis of a result of determining that the background complexity of an image to be processed is lower than a complexity threshold value, replacing an original background of the image to be processed with a template background to obtain a target image (S201); inputting the target image into a stylization processing model to obtain a preliminary stylization processed image (S202); and replacing the background of the preliminary stylization processed image with the original background to obtain a stylization processed image (S203).

Description

Image stylization processing method, device, electronic equipment and readable medium

This disclosure claims the priority of a Chinese patent application filed with the Chinese Patent Office with an application number of 201910910696.4 on September 25, 2019, and the entire content of the above application is incorporated into this disclosure by reference.

Technical field

The embodiments of the present application relate to the field of Internet technology, for example, to an image stylization processing method, device, electronic device, and readable medium.

Background technique

Most terminal devices in the related art are equipped with a photographing function. In order to meet the individual needs of terminal users, the photographing function of the terminal device usually has an image processing function to realize stylized processing of images captured by the terminal user.

In related technologies, when a terminal device performs stylization processing on an image, a method usually adopted is to directly perform stylization processing on the entire image through a neural network model and a stylization processing algorithm. However, because the content complexity of different images is different, it is difficult for the methods in related technologies to meet the stylization processing requirements of images of different complexity when actually performing stylization processing, and the stylization processing results are likely to be rough and affect the aesthetics. The situation of high degrees needs to be improved urgently.

Summary of the invention

The embodiments of the application provide an image stylization processing method, device, electronic device, and readable medium. When stylizing an image, the complexity of the image background is taken into consideration, and the fineness and precision of the image stylization processing result are improved. Aesthetics.

In the first aspect, an embodiment of the present application provides an image stylization processing method, which includes:

Based on the result of determining that the background complexity of the image to be processed is lower than the complexity threshold, replacing the original background of the image to be processed with the template background to obtain the target image;

Input the target image into a stylized processing model to obtain a preliminary stylized processed image;

The background of the preliminary stylized image is replaced with the original background to obtain the stylized image.

In the second aspect, an embodiment of the present application also provides an image stylization processing device, including:

The background replacement module is configured to replace the original background of the image to be processed with the template background based on the result of determining that the background complexity of the image to be processed is lower than the complexity threshold to obtain the target image;

The stylization processing module is set to input the target image into the stylization processing model to obtain a preliminary stylized processing image;

The background replacement module is further configured to replace the background of the preliminary stylized image with the original background to obtain the stylized image.

In a third aspect, an embodiment of the present application also provides an electronic device, which includes:

processor;

Memory, used to store programs;

When the program is executed by the processor, the processor realizes the image stylization processing method according to any embodiment of the present application.

In a fourth aspect, an embodiment of the present application provides a readable medium with a computer program stored on the readable medium, and when the computer program is executed by a processor, the image stylization process as described in any of the embodiments of the present application is realized. method.

Description of the drawings

With reference to the drawings and the following specific implementations, the above and other features, advantages, and aspects of the embodiments of the present application will become more apparent. Throughout the drawings, the same or similar reference signs indicate the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

Figures 1A-1B are schematic diagrams of the effect of stylizing images to be processed by related technologies;

2A is a flowchart of an image stylization processing method provided in Embodiment 1 of this application;

2B-2E are schematic diagrams of the effect of stylizing an image to be processed according to Embodiment 1 of the application;

FIG. 3 is a flowchart of an image stylization processing method provided in Embodiment 2 of the application;

4 is a flowchart of an image stylization processing method provided in Embodiment 3 of this application;

FIG. 5 is a schematic structural diagram of an image stylization processing apparatus provided in Embodiment 4 of the application;

FIG. 6 is a schematic structural diagram of an electronic device provided in Embodiment 5 of this application.

detailed description

Hereinafter, embodiments of the present application will be described in more detail with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein. These examples are provided for a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of the present application are only used for exemplary purposes, and are not used to limit the protection scope of the present application.

It should be understood that the steps described in the method embodiments of the present application may be executed in a different order, and/or executed in parallel. In addition, method implementations may include additional steps and/or omit to perform the illustrated steps. The scope of this application is not limited in this respect.

The term "including" and its variations as used herein are open-ended includes, that is, "including but not limited to". The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related definitions of other terms will be given in the following description.

It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules or units. Or interdependence.

It should be noted that the modification of “a” and “a plurality of” mentioned in this application is illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as “one or Multiple". The names of messages or information exchanged between multiple parties in the embodiments of the present application are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

It should be noted that, before introducing the embodiments of the present application, the applicable scenarios of the image stylization processing method, apparatus, electronic equipment, and readable medium of the embodiments of the present application are described first. The embodiments of this application are suitable for the process of stylizing images taken by users through electronic devices (such as mobile phones, tablets, wearable devices, cameras, etc.), and images stored in electronic device galleries, so as to generate personalization for users Image. When stylizing the image to be processed, a pre-trained neural network model is usually used to stylize the entire image. At this time, when the complexity of the background part of the image to be processed is relatively low, the stylization result is likely to be rough Case. Exemplarily, FIG. 1A is an image to be processed with a low background image complexity, and FIG. 1B is an effect diagram of the image to be processed shown in FIG. 1A after being processed in a comic style according to a method in the related art, and the overall effect of the effect diagram Poor, seriously affecting the aesthetics of manga style processing. Therefore, there is an urgent need to improve image stylization processing methods in related technologies.

The following describes in detail an image stylization processing method, device, electronic device, and readable medium provided in the following embodiments of the present application.

Example one

2A is a flowchart of an image stylization processing method provided in Embodiment 1 of this application; for example, 2B-2E are schematic diagrams of effects of stylizing an image to be processed provided in Embodiment 1 of this application. This embodiment is applicable to the case of stylizing the image to be processed, for example, it is applicable to the case of stylizing the image to be processed whose background complexity is lower than the complexity threshold. The method may be executed by an image stylization processing device or an electronic device, and the device may be implemented by software and/or hardware, and the device may be configured in the electronic device. Optionally, the electronic device may be a terminal device with image processing functions such as a mobile phone, a tablet computer, a wearable device, and a camera.

Optionally, as shown in FIG. 2A, the method in this embodiment may include the following steps:

S201: If it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with a template background to obtain a target image.

Among them, the image to be processed can be the image to be stylized in this embodiment, it can be an image currently shot by the camera on the electronic device, or it can be a saved image selected from the local gallery of the electronic device according to the user's click operation. . The original background of the image to be processed may be the original background of the image to be processed. For example, the original background of the image to be processed shown in FIG. 2B is a gray background. The template background is a pre-set high-complexity background image. Optionally, the number of template backgrounds in this embodiment can be one or more. When there are multiple template backgrounds, it can be set with multiple different complexities. The background image is used as the template background. The complexity threshold may be a pre-set criterion for judging whether to perform background replacement on the image to be processed. The complexity threshold may be a threshold set for the background complexity. The background complexity can be the complexity of the background area in the image to be processed. For example, when the background area of the image to be processed is a simple solid-color background such as a wall, blue sky, sea, etc., the background complexity is low; when the background area of the image to be processed When it is a chaotic background such as a road or a park, the background complexity is high.

Optionally, in this step, the first possible implementation manner for determining whether the background complexity of the image to be processed is lower than the complexity threshold is to perform pixel grayscale histogram statistics on the background area of the image to be processed, and the histogram is gray The wider the distribution range of the degree value, the higher the background complexity. The second possible implementation for determining whether the background complexity of the image to be processed is lower than the complexity threshold can be to extract the texture features of the original background of the image to be processed, and determine whether the background complexity of the image to be processed is Below the complexity threshold. In one embodiment, the implementation manner for determining whether the background complexity of the image to be processed is lower than the complexity threshold may be to first use a texture feature extraction algorithm to extract texture features from the original background of the image to be processed. Algorithms can include statistical methods, geometric methods, model methods, structural analysis methods, and signal processing methods. After extracting the texture feature of the original background of the image to be processed, the number of pixels corresponding to the texture feature or the ratio of the pixels corresponding to the texture feature to the total pixels of the original background can be used as the background complexity of the image to be processed, and then the Whether the background complexity is lower than the preset complexity threshold.

In this step, an optional execution method for executing the above-mentioned second implementable manner is: performing edge detection on the original background of the image to be processed to obtain the texture feature of the original background; determining that the texture feature is in the The proportion of pixels in the original background is used as the background complexity of the image to be processed; it is determined whether the background complexity of the image to be processed is lower than the complexity threshold. In an embodiment, the method may be to perform edge detection on the original background content of the image to be processed, and use the edge detection result as the texture feature of the original image. Among them, the edge detection algorithm may include canny algorithm, Roberts algorithm, Sobel algorithm and so on. Then determine the percentage of pixels corresponding to the edge result to the total number of pixels in the original background, as the background complexity of the image to be processed, compare the background complexity of the image to be processed with the preset complexity threshold, and determine Whether the background complexity of the image to be processed is lower than the complexity threshold.

Optionally, under normal circumstances, when stylizing the image to be processed, if the background complexity is low, the processing result will be rough. In this step, the background complexity of the image to be processed is higher than or equal to the complexity threshold, which can indicate that the background complexity of the image to be processed is high complexity. In this case, the uniform stylization processing method can accurately complete the image to be processed. The stylization processing operation is to directly input the image to be processed into the stylization processing model, and obtain the output result of the stylization processing model as the stylized image of the image to be processed. The background complexity of the image to be processed is lower than the complexity threshold, which can indicate that the background complexity of the image to be processed is relatively low, and the processing result is prone to be rough. The background of the image to be processed can be replaced, and the original background of the image to be processed can be replaced with The preset template background is used to obtain the target image. Exemplarily, FIG. 2B is the image to be processed in this embodiment. The original background of the image to be processed is a pure gray low-complexity background. If the background complexity of the image to be processed is lower than the complexity threshold, the The original gray background of FIG. 2B is replaced with a preset high-complexity template background of Peach Blossom Spring, to obtain the target image as shown in FIG. 2C.

Optionally, in this step, whether to determine the background complexity of the image to be processed or to perform a background replacement operation, the background area and the foreground area of the image to be processed may be determined first. In one embodiment, the determination method may be to use image recognition or image clustering algorithms to segment the image to be processed to obtain the background area and foreground area of the image to be processed; the determination method may also be to pre-train the background segmentation model, which is divided by the background. The model is used to segment the image to be processed to obtain the background area and foreground area of the image to be processed; the determination method can also be that the user manually selects the background area and foreground area of the image to be processed.

S202: Input the target image into the stylization processing model to obtain a preliminary stylization processing image.

Among them, the stylization processing model can be a pre-trained neural network model that can implement stylization processing on the content in the image. The training process of the stylized processing model will be described in detail in subsequent embodiments.

Optionally, in this step, the target image obtained after background replacement of the image to be processed in S201 can be used as the input parameter, and the program code of the pre-trained stylization processing model can be called and run. At this time, the stylization processing model will be based on The stylization processing algorithm during training stylizes the content in the input target image, and outputs the stylized image, that is, the preliminary stylized image.

Exemplarily, the target image shown in FIG. 2C is used as the input data of the stylization processing model, and the program code of the pre-trained stylization processing model is called and run. At this time, the stylization processing model can be based on the cartoon style during training. The processing algorithm is to perform comic stylization processing on the target image shown in FIG. 2C to obtain the preliminary stylized processed image shown in FIG. 2D.

Optionally, the stylization processing operation performed on the target image by the stylization processing model in this step may include: comic style processing, pixelized style processing, sketch style processing, oil painting style processing and other stylization processing in the form of multiple filters.

S203: Replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

Optionally, after the preliminary stylized image is obtained, since the background of the preliminary stylized image is not the original image of the image to be processed, in order to ensure the consistency between the processed image and the original image, the preliminary style may also be changed in this step. The current background of the transformed image is replaced with the original background of the image to be processed. In an embodiment, the execution process may be to first perform background area recognition on the preliminary stylized image, and determine the current background of the preliminary stylized image (in an embodiment, the determination method may be the same as determining the original background from the image to be processed). The process is the same, which will not be repeated), and the original background of the image to be processed is replaced with the current background of the preliminary stylized image to obtain the final stylized image after the image to be processed is stylized. Exemplarily, the current background of the preliminary stylized image shown in FIG. 2D is determined, and the original background of the image to be processed shown in FIG. 2B is replaced with the current background in FIG. 2D to obtain the stylized image shown in FIG. 2E (That is, the final stylized processed image).

Optionally, in this step, if the original background of the image to be processed is directly used to replace the background of the preliminary stylized image to generate the final stylized image, the background part of the finally obtained image does not perform the stylization operation. In order to improve the consistency and aesthetics of the stylization process of the image to be processed, in this step, when the background of the preliminary stylized image is replaced with the original background, the original background may also be input into the background processing model to obtain The processed original background; replacing the background of the preliminary stylized processed image with the processed original background. Among them, the background processing model is a pre-trained neural network model that specializes in stylizing the original background of the image to be processed. The type of stylization processing performed by the background processing model can be the same as the type of stylization processing performed by the stylization processing model in S202. The training process of the two models is also similar. The algorithm parameters of each model after training can be used when performing stylization processing. different. In an embodiment, the execution process may be to use the original background replaced by S201 as the input data of the background processing model, and call and run the program code of the background processing model. At this time, the background processing model will be based on the stylized processing algorithm during training. , Stylize the input original background to obtain the original stylized background, and then replace the background of the preliminarily stylized image with the original stylized background to obtain the final stylized image.

The embodiment of the application provides an image stylization processing method to determine whether the background complexity of the image to be processed is lower than the complexity threshold, and if so, replace the original background of the image to be processed with a preset template background to obtain the target Image: After the target image is stylized by the stylized processing model, the background of the processed preliminary stylized image is replaced with the original background to obtain the final stylized image. The solution of the embodiment of the application takes into account the complexity of the image background when stylizing the image. When the background complexity of the image to be processed is low, the method of replacing the template background for the image to be processed is used to perform stylization. This avoids the use of stylized processing methods in related technologies for images to be processed with low background complexity, and the processing results are rough, and the fineness and aesthetics of the image stylized processing results are improved.

Example two

Fig. 3 is a flowchart of the method for providing video learning materials provided in the second embodiment of the application; this embodiment has been modified on the basis of the optional solutions provided in the above-mentioned embodiments, and shows how to train the stylized processing model Detailed introduction.

Optionally, as shown in FIG. 3, the method in this embodiment may include the following steps:

S301, construct an initial network model.

Optionally, in this step, a convolutional neural network with a preset number of layers can be constructed in advance according to actual stylization processing requirements, and corresponding initial parameters can be set for each layer. The initial parameters can be the initial number of channels and loss weights. Values, channel parameters, stylization processing algorithm and its algorithm parameters, etc., to complete the construction of the initial network model. The initial network model constructed in this step cannot be directly used to stylize the image to be processed. You can first construct the initial network The model is trained.

S302, using the sample image data to perform stylization processing training on the initial network model.

Among them, the sample image data may be training data required for training the initial network model, and it may be composed of multiple sets of original images and stylized processed images of each original image.

Optionally, in this step, the stylized training of the initial network model constructed in S301 may be to sequentially use each set of original images in the sample image data and the stylized processed images of the original images as a set of input data, and input them to the initial network model. In the network model, the relevant initial parameters set in the initial network model are trained. In an embodiment, the training process may be similar to the training method of the neural network model that performs image stylization processing by the related technology.

S303: Use the verification image data to verify whether the attribute parameter of the output image of the initial network model after the training is greater than the parameter threshold, if not, execute S304, and if yes, execute S305.

The verification image data may be verification data used to verify whether the trained initial network model can perform stylized processing operations with high quality. Optionally, the verification image data can be obtained by using a certain proportion (such as 80%) of the image data in the acquired image as the sample image data, and the remaining proportion (such as 20%) of the image data as the verification image. Data; the verification image data can also be specially selected image data under various shooting scenes. The attribute parameter of the output image may be a judgment parameter used to verify the stylization processing effect of the output image, where the attribute parameter may include at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter, for each dimension of the attribute. The parameters can be determined by fixed formulas or algorithms.

Optionally, this step may be to use one or more sets of sample image data to stylize the initial network model, and then use the verification image data to verify whether the initial network model trained in S302 can perform stylization with high quality The processing operation is verified, that is, it is verified whether the initial network model trained in S302 can be used as a stylized processing model that can be actually used. In one embodiment, the verification process may be to use the verification image as the input data, call and run the program code of the trained initial network model. At this time, the trained initial network model will compare the input verification image according to the training algorithm. Perform stylization processing operations and output the stylized processing results, that is, verify the stylized processed image. Optionally, when determining the attribute parameters for the verified stylized image, the saturation of the image can be calculated as the saturation parameter; the smoothness of the edge contour in the image is calculated as the edge smoothness parameter; and the contours other than the edge contour of the image can be calculated. The smoothness of other areas is used as a shadow parameter. When judging whether the calculated attribute parameter is greater than the parameter threshold, the corresponding parameter threshold can be set for each parameter in the attribute parameter, and it is judged whether the various attribute parameters are greater than the corresponding parameter threshold. If each parameter value in the attribute parameter of the output image of the initial network model after training is greater than its corresponding parameter threshold, it means that the initial network model has been trained at this time, and S305 can be executed to use the trained initial network model as the Stylize the model; otherwise, it means that the effect of stylizing the initial network model at this time is not very good and can be further optimized. At this time, perform S304 to adjust and optimize the model parameters of the initial network model.

Optionally, in this embodiment, in order to ensure accurate verification of the trained initial network model, at least two sets of verification image data may be selected to perform this step on the trained initial network model. When the result of the verification of at least two sets of verification image data is that the attribute parameter of the output image is greater than the parameter threshold, S305 may be executed to use the trained initial network model as the stylization processing model.

S304: If not, adjust the model parameters of the initial network model within the preset modification range, and return to the initial network model after adjusting the model parameters to perform the operation of S302.

Optionally, when adjusting the model parameters of the initial network model trained in S302 in this step, at least one dimension of the number of layers for performing data processing operations within the model, the number of channels of each layer, and the loss weight value may be adjusted. Optionally, when adjusting the model parameters of the initial network model, you can set the priority for the number of layers, the number of channels, and the loss weight value. According to the priority, the model parameters of one dimension at a time are set in the corresponding model parameters. Adjust the model parameters within the preset modification range. After modifying the model parameters of one dimension, return to execute S302-S303, that is, use sample image data to re-stylize the initial network model after adjusting the model parameters, and use Verify the image data to verify whether the attribute parameters of the output image of the initial network model after training are greater than the parameter threshold. Optionally, if the priority of each model parameter is that the number of layers is higher than the number of channels, and the number of channels is higher than the loss weight value, you can adjust the number of channels first, increase the number of channels by 1, and then return to S302-S303, using sample images The data re-trains the initial network model after increasing the number of layers, and verifies whether the attribute parameters of the output image of the output initial network model after training are greater than the parameter threshold. If it is still not greater than the parameter threshold, then adjust the initial network model. After increasing the number of channels by 1.2 times, return to the operation of S302-S303. Optionally, when adjusting the number of channels of the initial network model in this step, the number of channels of the group convolutional layer of the initial network model may be adjusted first.

It should be noted that the preset modification range can be the first modification range set in advance, or at least two optional modification values set in advance. Optionally, for the loss weight value and the number of layers, one modification can be set Range: For the number of channels, you can set multiple optional modification values such as an increase of 1.2 times, 1.5 times, and 1.6 times.

Optionally, when adjusting the model parameters of the initial network model in this step, in addition to adjusting one model parameter at a time, at least two model parameters can also be adjusted at a time.

S305: If yes, use the trained initial network model as a stylized processing model.

Optionally, using verification image data to verify that the attribute parameters of the output image of the initial network model after training are greater than the parameter threshold, which can indicate that the initial network model training is completed at this time, and the program code of the initial network model at this time is encapsulated as stylized The program code to process the model.

S306: If it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image.

S307: Input the target image into the stylization processing model to obtain a preliminary stylized image.

S308: Replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

The embodiment of the application provides an image stylization processing method. After training the constructed initial network model using sample image data, if the verification image data is used to verify the multi-dimensionality of the stylized processed image output by the trained initial network model If the attribute parameter is not greater than the parameter threshold, the multi-dimensional model parameters of the initial network model are adjusted and retrained; otherwise, the training of the initial network model is completed, and the stylized processing model is obtained. In the process of training the initial network model in this embodiment, through the verification of the multi-dimensional attribute parameters and the adjustment of the multi-dimensional model parameters, the quality of the processing effect of the stylization processing model is greatly improved, and the stylization of the subsequent processing image The processing operation lays the foundation. When stylizing the image to be processed, if the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the preset template background, and then use the stylized processing model to style The final stylized image is obtained after the background of the obtained preliminary stylized image is replaced with the original background. The background complexity of the image to be processed is considered, and the fineness and precision of the image stylized processing result are further improved. Aesthetics.

Example three

4 is a flowchart of a method for providing video learning materials provided in Embodiment 3 of the application; this embodiment has been modified on the basis of the optional solutions provided in the foregoing embodiments, and the input style of the target image is given In the stylized processing model, a detailed introduction to the preliminary stylized image is obtained.

Optionally, as shown in FIG. 4, the method in this embodiment may include the following steps:

S401: If it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with a template background to obtain a target image.

S402: Determine an area to be processed according to the received area selection instruction.

Optionally, in this embodiment, in order to improve the user's individual requirements for stylizing the image to be processed, it may be based on the user's selection to perform stylization only on the user's selection area. In one embodiment, after the user selects the image to be processed, a sliding operation on the image to be processed triggers an area selection instruction, where the area selection instruction includes the area to be processed selected by the user. Optionally, if the region selection instruction is triggered by the user's non-closed sliding operation on the image to be processed, the region corresponding to the user's sliding operation may be used as the region to be processed, and the position coordinates corresponding to the region to be processed Is added to the area selection instruction and transmitted to the electronic device; if the area selection instruction is triggered by the user's closed sliding operation on the image to be processed, the closed frame selection area corresponding to the user's sliding operation can be used as the area to be processed, and The position coordinates corresponding to the area to be processed are added to the area selection instruction and transmitted to the electronic device. After receiving the area selection instruction triggered by the user, the electronic device obtains the coordinate position of the area to be processed from the area selection instruction.

S403: Input the target image and the region to be processed into a stylization processing model, and control the stylization processing model to perform stylization processing on the content of the region to be processed in the target image to obtain a preliminary stylized image.

Optionally, in this step, the target image obtained in S401 and the region to be processed determined in S402 can be used as input data, and the program code of the stylized processing model is called and executed. At this time, the stylized processing model is based on the input target image and For the area to be processed, according to the training algorithm, only the content of the area to be processed in the target image is stylized. In the preliminary stylized image obtained, only the area to be processed is displayed with the effect of stylization, and the rest of the area Still not stylized.

S404: Replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

Optionally, if the area to be processed determined in S402 does not include a background area, this step may be to replace the background of the initial stylized image obtained in S403 with the original background of the image to be processed to obtain the final stylized image. If the area to be processed determined in S402 contains the background area, this step may be to determine the position coordinates of the area to be processed in the background area, and then use the original background and the position coordinates of the area to be processed in the background area as the input data. In the background processing model, the background processing model is controlled to only stylize the content of the position coordinate area in the original background to obtain the processed original background; finally, the background of the initial stylized image obtained in S403 is replaced with the processed original Background, get the final stylized image.

The embodiment of the application provides an image stylization processing method. If a user triggers a region selection instruction when selecting an image to be processed, when the background complexity of the image to be processed is lower than the complexity threshold, the The original background is replaced with the preset template background. After the target image is obtained, the area to be processed is determined according to the area selection instruction triggered by the user, and the stylization processing model is controlled to only stylize the area to be processed in the target image. The background of the preliminary stylized image is replaced with the original background to obtain the final stylized image. When stylizing the image to be processed, the solution of the embodiment of the present application not only considers the complexity of the image background, but also improves the fineness and beauty of the image stylization processing result; it can also generate the user's region selection instructions for the user The personalized stylized processing effect improves the fun of image stylized processing.

Example four

FIG. 5 is a schematic structural diagram of an image stylization processing device provided in Embodiment 4 of the application. The embodiment of this application can be applied to a situation where an image to be processed is stylized, for example, when the background complexity is lower than the complexity threshold. Deal with the situation where the image is stylized. The device can be implemented by software and/or hardware, and integrated in the electronic device that executes the method. As shown in Fig. 5, the device can include:

The background replacement module 501 is configured to, if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image;

The stylization processing module 502 is configured to input the target image into the stylization processing model to obtain a preliminary stylized processing image;

The background replacement module 501 is further configured to replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

The embodiment of the application provides an image stylization processing device to determine whether the background complexity of the image to be processed is lower than the complexity threshold, and if so, replace the original background of the image to be processed with a preset template background to obtain the target Image: After the target image is stylized by the stylized processing model, the background of the processed preliminary stylized image is replaced with the original background to obtain the final stylized image. The solution of the embodiment of the application takes into account the complexity of the image background when stylizing the image. When the background complexity of the image to be processed is low, the method of replacing the template background for the image to be processed is used to perform stylization. This avoids the use of stylized processing methods in related technologies for images to be processed with low background complexity, and the processing results are rough, and the fineness and aesthetics of the image stylized processing results are improved.

In an embodiment, the device further includes a complexity judgment model, and the complexity judgment model is set as follows when the background complexity of the image to be processed is determined to be lower than the complexity threshold value:

The texture feature of the original background of the image to be processed is extracted, and according to the texture feature, it is determined whether the background complexity of the image to be processed is lower than the complexity threshold.

In an embodiment, the complexity judgment model may be set as:

Performing edge detection on the original background of the image to be processed to obtain the texture feature of the original background;

Determining the proportion of pixels of the texture feature in the original background as the background complexity of the image to be processed;

It is determined whether the background complexity of the image to be processed is lower than the complexity threshold.

In an embodiment, the device further includes: a model training module, and the model training module includes:

The training unit is set to use sample image data to stylize the constructed initial network model;

The verification unit is configured to use the verification image data to verify whether the attribute parameter of the output image of the initial network model after training is greater than the parameter threshold; wherein the attribute parameter includes: at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter ；

The model determining unit is configured to use the trained initial network model as the stylization processing model if the attribute parameter of the output image of the initial network model after training is greater than the parameter threshold.

In an embodiment, the model training module further includes:

The parameter adjustment unit is configured to adjust the model parameters of the initial network model within a preset modification range if the attribute parameters of the output image of the initial network model after training are less than or equal to the parameter threshold; wherein, the model parameters include the initial At least one of the number of layers, the number of channels, and the loss weight value of the network model;

The training unit is also configured to use sample image data to re-train the initial network model after adjusting the model parameters.

In an embodiment, the stylization processing module 502 may be configured as:

Determine the area to be processed according to the received area selection instruction;

The target image and the region to be processed are input into a stylization processing model, and the stylization processing model is controlled to perform stylization processing on the content of the region to be processed of the target image to obtain a preliminary stylized image.

In an embodiment, when the background replacement module 501 executes to replace the background of the preliminary stylized image with the original background, it may be set to:

Input the original background into the background processing model to obtain the processed original background;

The background of the preliminary stylized processed image is replaced with the processed original background.

The image stylization processing device provided in the embodiments of this application belongs to the same inventive concept as the image stylization processing methods provided in the above embodiments. For technical details not described in the embodiments of this application in detail, please refer to the above embodiments. The application embodiments have the same beneficial effects as the foregoing embodiments.

Example five

Next, referring to FIG. 6, FIG. 6 is a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present application. The electronic device may include mobile phones such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), etc. Terminals and fixed terminals such as digital TVs, desktop computers, etc. The electronic device 600 shown in FIG. 6 is only an example.

As shown in FIG. 6, the electronic device 600 may include a processing device (such as a central processing unit, a graphics processor, etc.) 601, which may be loaded into a random access device according to a program stored in a read-only memory (ROM) 602 or from a storage device 608. The program in the memory (RAM) 603 executes various appropriate actions and processing. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to the bus 604.

Generally, the following devices can be connected to the I/O interface 605: including input devices 606 such as touch screens, touch pads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; including, for example, liquid crystal displays (LCD), speakers, vibration An output device 607 such as a device; a storage device 608 such as a magnetic tape and a hard disk; and a communication device 609. The communication device 609 may allow the electronic device 600 to perform wireless or wired communication with other devices to exchange data. Although FIG. 6 shows an electronic device 600 having various devices, it should be understood that it is not required to implement or have all the illustrated devices. It may alternatively be implemented or provided with more or fewer devices.

According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions in the method of the embodiment of the present application are executed.

It should be noted that the above-mentioned computer-readable medium in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the above. Examples of computer-readable storage media may include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In this application, the computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, and can include electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable signal medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, which may include: wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.

In some embodiments, the electronic device can communicate with any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with any form or medium of digital data ( For example, communication networks) are interconnected. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (for example, the Internet), and end-to-end networks (for example, ad hoc end-to-end networks), as well as any currently known or future research and development network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device.

The aforementioned computer-readable medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the internal processes of the electronic device are executed: if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, Then replace the original background of the image to be processed with the template background to obtain the target image; input the target image into the stylization processing model to obtain the preliminary stylized image; replace the background of the preliminary stylized image with The original background obtains the final stylized processed image.

The computer program code used to perform the operations of the present application can be written in one or more programming languages or a combination thereof. The aforementioned programming languages can include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional The procedural programming language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to pass Internet connection).

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation of the system architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more for realizing the specified logical function Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations Or it can be realized by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present application can be implemented in software or hardware. Among them, the name of the unit does not constitute a limitation on the unit itself.

The functions described hereinabove may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logical device (CPLD) and so on.

In the context of the present application, a machine-readable medium may be a tangible medium, which may contain or store a program for use by the instruction execution system, apparatus, or device or in combination with the instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. Examples of machine-readable storage media may include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), optical fiber, compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

According to an image stylization processing method provided by one or more embodiments of the present application, the method includes:

If it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image;

The background of the preliminary stylized image is replaced with the original background to obtain the final stylized image.

According to one or more embodiments of the present application, in the foregoing method, determining that the background complexity of the image to be processed is lower than the complexity threshold includes:

According to one or more embodiments of the present application, in the above method, extracting the texture feature of the original background of the image to be processed, and determining whether the background complexity of the image to be processed is lower than the complexity threshold according to the texture feature includes:

According to one or more embodiments of the present application, in the above method, before inputting the target image into the stylization processing model, the method further includes:

Use sample image data to stylize the constructed initial network model;

The verification image data is used to verify whether the attribute parameters of the output image of the initial network model after the training are greater than the parameter threshold; wherein, the attribute parameters include at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter;

If so, the initial network model after training is used as the stylized processing model.

According to one or more embodiments of the present application, in the above method, after verifying whether the attribute parameter of the output image of the initial network model after training is greater than the parameter threshold, the method further includes:

If not, adjust the model parameters of the initial network model within a preset modification range; wherein, the model parameters include at least one of the number of layers, the number of channels, and the loss weight value of the initial network model;

The sample image data is used to re-train the initial network model after adjusting the model parameters.

According to one or more embodiments of the present application, in the above method, inputting the target image into a stylization processing model to obtain a preliminary stylization processing image includes:

According to one or more embodiments of the present application, in the foregoing method, replacing the background of the preliminary stylized image with the original background includes:

An image stylization processing device provided according to one or more embodiments of the present application includes:

The background replacement module is configured to, if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image;

The background replacement module is further configured to replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

According to one or more embodiments of the present application, the above-mentioned apparatus further includes a complexity judgment model, which is set as follows when the background complexity of the image to be processed is determined to be lower than the complexity threshold value:

According to one or more embodiments of the present application, the complexity judgment model in the above-mentioned device may be set as:

According to one or more embodiments of the present application, the above-mentioned device further includes: a model training module, and the model training module includes:

According to one or more embodiments of the present application, the model training module in the above-mentioned device further includes:

According to one or more embodiments of the present application, the stylization processing module in the above-mentioned device may be set to:

According to one or more embodiments of the present application, when the background replacement module in the above-mentioned device executes the replacement of the background of the preliminary stylized image with the original background, it may be set to:

According to an electronic device provided by one or more embodiments of the present application, the electronic device includes:

One or more processors;

The memory is set to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the image stylization processing method according to any embodiment of the present application.

A readable medium provided according to one or more embodiments of the present application has a computer program stored thereon, and when the program is executed by a processor, the image stylization processing method as described in any of the embodiments of the present application is implemented.

Although the operations are depicted in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, although implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Some features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or logical actions of the method, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely exemplary forms of implementing the claims.

Claims

An image stylization processing method, including:

Based on the result of determining that the background complexity of the image to be processed is lower than the complexity threshold, replacing the original background of the image to be processed with the template background to obtain the target image;

Input the target image into a stylized processing model to obtain a preliminary stylized processed image;

The background of the preliminary stylized image is replaced with the original background to obtain the stylized image.
The method according to claim 1, further comprising:

Determine whether the background complexity of the image to be processed is lower than the complexity threshold;

Among them, determining whether the background complexity of the image to be processed is lower than the complexity threshold includes:

The texture feature of the original background of the image to be processed is extracted, and according to the texture feature, it is determined whether the background complexity of the image to be processed is lower than the complexity threshold.
The method according to claim 2, wherein extracting the texture feature of the original background of the image to be processed, and determining whether the background complexity of the image to be processed is lower than the complexity threshold according to the texture feature, comprises:

Performing edge detection on the original background of the image to be processed to obtain the texture feature of the original background;

Determining the proportion of pixels of the texture feature in the original background as the background complexity of the image to be processed;

It is determined whether the background complexity of the image to be processed is lower than the complexity threshold.
The method according to claim 1, before inputting the target image into the stylization processing model, the method further comprises:

Use sample image data to stylize the constructed initial network model;

The verification image data is used to verify whether the attribute parameters of the output image of the initial network model after the training are greater than the parameter threshold; wherein, the attribute parameters include at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter;

Based on the verification result that the attribute parameter of the output image of the initial network model after training is greater than the parameter threshold, the initial network model after training is used as the stylization processing model.
The method according to claim 4, wherein, after verifying whether the attribute parameters of the output image of the initial network model after the training are greater than the parameter threshold using the verification image data, the method further comprises:

Based on the verification result that the attribute parameters of the output image of the initial network model after training are not greater than the parameter threshold, the model parameters of the initial network model are adjusted within a preset modification range; wherein, the model parameters include the number of layers of the initial network model, At least one of the number of channels and loss weight value;

Using the sample image data to re-train the initial network model after adjusting the model parameters for stylization processing.
The method according to claim 1, wherein inputting the target image into a stylization processing model to obtain a preliminary stylization processing image comprises:

Determine the area to be processed according to the received area selection instruction;

The target image and the region to be processed are input into a stylization processing model, and the stylization processing model is controlled to perform stylization processing on the content of the region to be processed of the target image to obtain a preliminary stylized image.
The method according to claim 1, wherein replacing the background of the preliminary stylized image with the original background comprises:

Input the original background into a background processing model to obtain a processed original background;

The background of the preliminary stylized processed image is replaced with the processed original background.
An image stylization processing device, including:

The background replacement module is configured to replace the original background of the image to be processed with the template background based on the result of determining that the background complexity of the image to be processed is lower than the complexity threshold to obtain the target image;

The stylization processing module is set to input the target image into the stylization processing model to obtain a preliminary stylized processing image;

The background replacement module is further configured to replace the background of the preliminary stylized image with the original background to obtain the stylized image.
An electronic device including:

processor;

Memory, used to store programs;

When the program is executed by the processor, the processor realizes the image stylization processing method according to any one of claims 1-7.
A readable medium on which a computer program is stored, and when the computer program is executed by a processor, the image stylization processing method according to any one of claims 1-7 is realized.