CN110689478B - Image stylization processing method and device, electronic equipment and readable medium - Google Patents

Abstract

The disclosure discloses an image stylization processing method, an image stylization processing device, electronic equipment and a readable medium. The method comprises the following steps: if the background complexity of the image to be processed is determined to be lower than the complexity threshold, replacing the original background of the image to be processed with a template background to obtain a target image; inputting the target image into a stylized processing model to obtain a preliminary stylized processing image; and replacing the background of the preliminary stylized image with the original background to obtain a final stylized image. According to the scheme of the embodiment of the disclosure, when the image is stylized, the complexity of the image background is considered, and the fineness and the attractiveness of the image stylized result are improved.

Description

Image stylization processing method, device, electronic equipment and readable medium

Technical field

The embodiments of the present disclosure relate to the field of Internet technology, and in particular, to an image stylization processing method, device, electronic device, and readable medium.

Background technique

Most of the existing terminal devices are equipped with a camera function. In order to meet the personalized needs of end users, the camera function of the terminal device usually has an image processing function to stylize the images captured by the end user.

Currently, when terminal devices stylize images, they use a stylization algorithm through a neural network model to directly stylize the entire image. However, because the content complexity of different images is different, it is difficult for existing methods to meet the stylization needs of images of different complexity when actually performing stylization processing, and it is easy to produce rough stylization results that affect the aesthetics. The situation needs to be improved urgently.

Contents of the invention

Embodiments of the present disclosure provide an image stylization processing method, device, electronic device, and readable medium. When stylizing an image, the complexity of the image background is taken into consideration, thereby improving the precision and accuracy of the image stylization processing results. Aesthetics.

In a first aspect, embodiments of the present disclosure provide an image stylization processing method, which method includes:

If it is determined that the background complexity of the image to be processed is lower than the complexity threshold, the original background of the image to be processed is replaced with the template background to obtain the target image;

Input the target image into the stylized processing model to obtain a preliminary stylized image;

The background of the preliminary stylized image is replaced with the original background to obtain a final stylized image.

In a second aspect, embodiments of the present disclosure also provide an image stylization processing device, which is characterized in that it includes:

A background replacement module, used to replace the original background of the image to be processed with the template background to obtain the target image if it is determined that the background complexity of the image to be processed is lower than the complexity threshold;

A stylization processing module, used to input the target image into the stylization processing model to obtain a preliminary stylization processing image;

The background replacement module is also used to replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

In a third aspect, embodiments of the present disclosure also provide an electronic device, which includes:

one or more processors;

Memory, used to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the image stylization processing method described in any embodiment of the present disclosure.

In a fourth aspect, embodiments of the present disclosure provide a readable medium on which a computer program is stored. When the program is executed by a processor, the image stylization processing method as described in any embodiment of the present disclosure is implemented.

Embodiments of the present disclosure provide an image stylization processing method, device, electronic device and readable medium to determine whether the background complexity of the image to be processed is lower than the complexity threshold, and if so, replace the original background of the image to be processed As the preset template background, the target image is obtained; after the stylization processing model stylizes the target image, the background of the preliminary stylized image obtained is replaced with the original background to obtain the final stylized image. The solution of the embodiment of the present disclosure takes the complexity of the image background into consideration when stylizing the image. When the background complexity of the image to be processed is low, the method of changing the template background for the image to be processed and then performing stylization processing is adopted. , which avoids the rough processing results caused by the existing stylization processing method for images to be processed with low background complexity, and improves the precision and beauty of the image stylization processing results.

Description of the drawings

The above and other features, advantages, and aspects of various embodiments of the present disclosure will become more apparent with reference to the following detailed description taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It is to be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

Figures 1A-1B show a schematic diagram of the effect of stylizing the image to be processed using the prior art;

Figure 2A shows a flow chart of an image stylization processing method provided by Embodiment 1 of the present disclosure;

2B-2E are schematic diagrams showing the effect of stylizing the image to be processed provided by Embodiment 1 of the present disclosure;

Figure 3 shows a flow chart of another image stylization processing method provided by Embodiment 2 of the present disclosure;

Figure 4 shows a flow chart of another image stylization processing method provided by Embodiment 3 of the present disclosure;

Figure 5 shows a schematic structural diagram of an image stylization processing device provided in Embodiment 4 of the present disclosure;

FIG. 6 shows a schematic structural diagram of an electronic device provided in Embodiment 5 of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, which rather are provided for A more thorough and complete understanding of this disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

It should be understood that various steps described in the method implementations of the present disclosure may be executed in different orders and/or in parallel. Furthermore, method embodiments may include additional steps and/or omit performance of illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "include" and its variations are open-ended, ie, "including but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as “first” and “second” mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units. Or interdependence.

It should be noted that the modifications of "one" and "plurality" mentioned in this disclosure are illustrative and not restrictive. Those skilled in the art will understand that unless the context clearly indicates otherwise, it should be understood as "one or Multiple”. The names of messages or information exchanged between multiple parties in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

It should be noted that before introducing the embodiments of the present disclosure, the applicable scenarios of the image stylization processing methods, devices, electronic devices and readable media according to the embodiments of the present disclosure will be described. Specifically, the embodiments of the present disclosure are applicable to the process of stylizing images taken by users through electronic devices (such as mobile phones, tablets, wearable devices, cameras, etc.) and images already stored in the electronic device gallery, so as to generate images for the user. Personalized images. When stylizing the image to be processed, the pre-trained neural network model usually stylizes the entire image. At this time, when the complexity of the background part of the image to be processed is relatively low, the stylization result is prone to be rough. Condition. For example, Figure 1A is an image to be processed with a low background image complexity. Figure 1B is a rendering of the image to be processed shown in Figure 1A after being processed in a comic style according to the existing method. The overall effect of the rendering is poor. , seriously affecting the aesthetics of comic-style processing. Therefore, there is an urgent need to improve existing image stylization processing methods.

An image stylization processing method, device, electronic device, and readable medium provided by the following embodiments of the present disclosure will be described in detail below.

Embodiment 1

Figure 2A shows a flow chart of an image stylization processing method provided in Embodiment 1 of the present disclosure; Figures 2B-2E show schematic diagrams of the effect of stylizing an image to be processed provided in Embodiment 1 of the present disclosure. This embodiment can be applied to the situation of stylizing the image to be processed, and is particularly suitable to the situation of stylizing the image to be processed whose background complexity is lower than the complexity threshold. The method can be performed by an image stylization processing device or an electronic device. The device can be implemented by software and/or hardware. The device can be configured in the electronic device. Optionally, the electronic device may be a terminal device with image processing functions such as a mobile phone, a tablet computer, a wearable device, and a camera.

Optionally, as shown in Figure 2A, the method in this embodiment may include the following steps:

S201, if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image.

The image to be processed may be the image to be stylized in this embodiment, it may be the image currently captured by the camera on the electronic device, or it may be an existing image selected from the local gallery of the electronic device according to the user's click operation. image. The original background of the image to be processed may be the original background of the image to be processed. For example, the original background of the image to be processed shown in Figure 2B is a gray background. The template background is a preset high-complexity background image. Optionally, the number of template backgrounds in this embodiment can be one or more. When there are multiple template backgrounds, multiple template backgrounds can be set with different complexity. background image as template background. The complexity threshold may be a preset criterion used to determine whether to perform background replacement on the image to be processed. The complexity threshold may be a threshold set for background complexity. Background complexity can be the complexity of the background area in the image to be processed. For example, when the background area of the image to be processed is a simple solid color background such as a wall, blue sky, sea, etc., the background complexity is low; when the background area of the image to be processed is When it is a cluttered background such as a road or a park, the background complexity is high.

Optionally, in this step, an implementation method for determining whether the background complexity of the image to be processed is lower than the complexity threshold may be to perform pixel grayscale histogram statistics on the background area of the image to be processed. If the gray level in the histogram is The wider the distribution range of degree values, the higher the background complexity. The second possible implementation may be to extract texture features of the original background of the image to be processed, and determine whether the background complexity of the image to be processed is lower than a complexity threshold based on the texture features. Specifically, a texture feature extraction algorithm may be first used to extract texture features from the original background of the image to be processed. The algorithm for extracting texture features may include but is not limited to statistical methods, geometric methods, model methods, structural analysis methods, and signal processing. Law etc. After extracting the texture features of the original background of the image to be processed, the number of pixels corresponding to the texture features or the ratio of the pixels corresponding to the texture features to the total pixels of the original background can be used as the background complexity of the image to be processed, and then the Whether the background complexity is lower than the preset complexity threshold.

In a preferred embodiment, a preferred execution method of this step when executing the above second implementation mode may be: performing edge detection on the original background of the image to be processed to obtain the texture characteristics of the original background; determining the The pixel ratio of the texture feature in the original background is used as the background complexity of the image to be processed; it is determined whether the background complexity of the image to be processed is lower than the complexity threshold. Specifically, the method may be to perform edge detection on the original background content of the image to be processed, and use the edge detection results as texture features of the original image. Among them, edge detection algorithms may include but are not limited to canny algorithm, Roberts algorithm, Sobel algorithm, etc. Then determine the pixel ratio of the number of pixels corresponding to the edge result to the total number of pixels in the original background, as the background complexity of the image to be processed, compare it with the preset complexity threshold, and determine the background complexity of the image to be processed. Is the degree below the complexity threshold?

Optional. Usually, when stylizing the image to be processed, if the background complexity is low, the processing results will be rough. Therefore, this step can indicate that the background complexity of the image to be processed is high complexity when the background complexity of the image to be processed is higher than or equal to the complexity threshold. At this time, a unified stylization processing method can be used to accurately complete the image to be processed. Stylization processing operation, that is, you can directly input the image to be processed into the stylization processing model, and obtain the output result of the stylization processing model as the stylized processing image of the image to be processed. When the background complexity of the image to be processed is lower than the complexity threshold, it means that the background complexity of the image to be processed is relatively low, and rough processing results are prone to occur. The background of the image to be processed needs to be replaced, and the original background of the image to be processed is replaced. Get the target image for the preset template background. Exemplarily, Figure 2B shows an image to be processed in this embodiment. The original background of the image to be processed is a pure gray low-complexity background. If the background complexity of the image to be processed is lower than the complexity threshold, it can be The gray original background in Figure 2B is replaced with the preset high-complexity Peach Blossom Spring template background, and the target image shown in Figure 2C is obtained.

Optionally, whether this step is to determine the background complexity of the image to be processed or to perform a background replacement operation, it is necessary to first determine the background area and foreground area of the image to be processed. The specific determination method may be to use image recognition or image clustering. The algorithm performs area segmentation on the image to be processed to obtain the background area and foreground area of the image to be processed; it can also pre-train the background segmentation model, and use the background segmentation model to segment the area on the image to be processed to obtain the background area and foreground area of the image to be processed. ; It is also possible for the user to manually select the background area and foreground area of the image to be processed. This embodiment is not limited to this.

S202: Input the target image into the stylized processing model to obtain a preliminary stylized image.

Among them, the stylization processing model can be pre-trained, and can implement a neural network model that stylizes the content in the image. The specific training process of this stylized processing model will be introduced in detail in subsequent embodiments.

Optionally, this step can be to use the target image obtained after background replacement of the image to be processed in S201 as an input parameter, and call and run the program code of the pre-trained stylized processing model. At this time, the stylized processing model will be based on the training The current stylization algorithm stylizes the content in the input target image and outputs the stylized image, which is the preliminary stylized image.

For example, the target image shown in Figure 2C is used as the input data of the stylization processing model, and the program code of the pre-trained stylization processing model is called and run. At this time, the stylization processing model will be based on the comic style during training. The processing algorithm performs comic stylization on the target image shown in Figure 2C to obtain the preliminary stylized image shown in Figure 2D.

Optionally, the stylization processing operation performed on the target image by the stylization processing model in this step may include but is not limited to: comic style processing, pixelation style processing, sketch style processing or oil painting style processing and other styles in the form of filters. chemical treatment.

S203: Replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

Optionally, after obtaining the preliminary stylized image, since the background of the preliminary stylized image is not the original image of the image to be processed, in order to ensure the consistency between the processed image and the original image, this step also needs to change the preliminary style Replaces the current background of the image with the original background of the image to be processed. The specific execution process may be to first perform background area recognition on the preliminary stylized image, and determine the current background of the preliminary stylized image (the specific determination method is the same as the process of determining the original background from the image to be processed, and will not be described in detail. ), and replace the current background of the preliminary stylized image with the original background of the image to be processed, to obtain the final stylized image that is stylized on the image to be processed. Exemplarily, the current background of the preliminary stylized image shown in Figure 2D is determined, and the original background of the image to be processed shown in Figure 2B is replaced with the current background in Figure 2D to obtain the final stylized image shown in Figure 2E Process images.

Optionally, if this step directly uses the original background of the image to be processed to replace the background of the preliminary stylized image to generate the final stylized image, the background part of the final image will not be stylized. In order to improve the consistency and aesthetics of the stylized image to be processed, when performing this step of replacing the background of the preliminary stylized image with the original background, the original background can also be input into the background processing model, and we obtain The processed original background; replace the background of the preliminary stylized image with the processed original background. Among them, the background processing model is a pre-trained neural network model that specializes in stylizing the original background of the image to be processed. The type of stylization processing performed by this background processing model is the same as the type of stylization processing performed by the stylization processing model in S202. The training process of the two models is also similar, except that the algorithm parameters of each model after training are used to specifically perform stylization processing. different. The specific execution process can be to use the original background replaced by S201 as the input data of the background processing model, and call and run the program code of the background processing model. At this time, the background processing model will process the input based on the stylized processing algorithm during training. The original background is stylized to obtain the stylized original background, and then the background of the preliminary stylized image is replaced with the stylized original background to obtain the final stylized image.

Embodiments of the present disclosure provide an image stylization processing method to determine whether the background complexity of the image to be processed is lower than the complexity threshold. If so, replace the original background of the image to be processed with a preset template background to obtain the target Image; after the target image is stylized by the stylization processing model, the background of the preliminary stylized image obtained is replaced with the original background to obtain the final stylized image. The solution of the embodiment of the present disclosure takes the complexity of the image background into consideration when stylizing the image. When the background complexity of the image to be processed is low, the method of changing the template background for the image to be processed and then performing stylization processing is adopted. , which avoids the rough processing results caused by the existing stylization processing method for images to be processed with low background complexity, and improves the precision and beauty of the image stylization processing results.

Embodiment 2

Figure 3 shows a flow chart of another method for providing video learning materials provided by Embodiment 2 of the present disclosure; this embodiment is optimized based on each option provided by the above embodiment, and specifically provides how Detailed introduction to training stylization processing models.

Optionally, as shown in Figure 3, the method in this embodiment may include the following steps:

S301. Build an initial network model.

Optionally, this step can pre-construct a convolutional neural network with a preset number of layers according to the actual stylization processing requirements, and set corresponding initial parameters for each layer, where the initial parameters can be the initial number of channels and loss weight. values, channel parameters, stylization processing algorithm and its algorithm parameters, etc., to complete the construction of the initial network model. The initial network model constructed in this step cannot be directly used to stylize the image to be processed. The constructed initial network needs to be first The model is trained.

S302: Use the sample image data to perform stylized processing training on the initial network model.

The sample image data may be training data required to train the initial network model, which may be composed of multiple sets of original images and stylized images of each original image.

Optionally, in this step, when performing stylized training on the initial network model constructed by S301, each set of original images and stylized images of the original images in the sample image data can be used as a set of input data and input into In the initial network model, the relevant initial parameters set in the initial network model are trained. The specific training process may be similar to the training method of the neural network model that performs image stylization processing in the prior art, which is not limited in this embodiment.

S303. Use the verification image data to verify whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold. If not, perform S304. If yes, perform S305.

The verification image data may be verification data used to verify whether the trained initial network model can perform stylization processing operations with high quality. Optionally, verifying the image data can be during the process of obtaining the sample image, using a certain proportion (such as 80%) of the image data in the obtained image as the sample image data, and the remaining proportion (such as 20%) of the image data as the verification image. Data; it can also be specially selected image data under various shooting scenarios. The attribute parameters of the output image may be evaluation parameters used to verify the stylization processing effect of the output image, wherein the attribute parameters may include but are not limited to at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter, for each of which The attribute parameters of dimensions can be determined through fixed formulas or algorithms.

Optionally, this step can be to use one or more sets of sample image data to stylize and train the initial network model, and then use the verification image data to determine whether the initial network model trained in S302 can perform stylization with high quality. The processing operation is verified, that is to say, it is verified whether the initial network model obtained by S302 training can be used as a stylized processing model that can be actually used. The specific verification process can be to use the verification image as input data, call and run the program code of the initial network model after training, and then the initial network model after training will stylize the input verification image according to the algorithm during training. Operation, output the stylized processing result, that is, verify the stylized image. Optionally, when determining the attribute parameters for the verification stylized image, you can calculate the saturation of the image as the saturation parameter; calculate the smoothness of the edge contour in the image as the edge smoothness parameter; calculate the edge contour other than the image. The smoothness of other areas is used as a shading parameter. When determining whether the calculated attribute parameters are greater than the parameter threshold, a corresponding parameter threshold can be set for each parameter in the attribute parameters to determine whether each attribute parameter is greater than its corresponding parameter threshold. If each parameter value in the attribute parameters of the output image of the trained initial network model is greater than its corresponding parameter threshold, it means that the initial network model has been trained at this time, and S305 can be executed to use the trained initial network model as the above Stylized processing model. Otherwise, it means that the stylization effect of the initial network model at this time is not very good and further optimization is needed. At this time, S304 is executed to adjust and optimize the model parameters of the initial network model.

Optionally, in this embodiment, in order to ensure accurate verification of the initial network model after training, at least two sets of verification image data may be selected to perform this step on the initial network model after training. When the verification results of at least two sets of verification image data are that the attribute parameters of the output image are greater than the parameter threshold, S305 is executed, and the trained initial network model is used as the stylization processing model.

S304, if not, adjust the model parameters of the initial network model within the preset modification range, and return to perform the operation of S302 on the initial network model after adjusting the model parameters.

Optionally, in this step, when adjusting the model parameters of the initial network model after training in S302, at least one dimension of the number of layers performing data processing operations within the model, the number of channels in each layer, and the loss weight value can be adjusted. Optionally, when adjusting the model parameters of the initial network model, you can set the priority for the number of layers, the number of channels, and the loss weight value. According to the priority, for each dimension of the model parameters, in the corresponding model parameter Adjust the model parameters within the preset modification range. After modifying the model parameters of one dimension, return to execute S302-S303. Use sample image data to re-stylize and train the initial network model after adjusting the model parameters, and use verification Image data verifies whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold. Preferably, if the priority of each model parameter is that the number of layers is higher than the number of channels, and the number of channels is higher than the loss weight value, you can first adjust the number of channels, increase the number of channels by 1, and then return to execute S302-S303, using the sample image data Re-train the initial network model after adding layers, and verify whether the attribute parameters of the output image of the trained output initial network model are greater than the parameter threshold. If they are still not greater than the parameter threshold, then adjust the initial network model. The number of channels, increase the number of channels by 1.2 times and then return to perform operations S302-S303. Optionally, when adjusting the number of channels of the initial network model in this step, you may first choose to adjust the number of channels of the group convolution layer of the initial network model.

It should be noted that the preset modification range can be the first modification range set in advance; it can also be at least two optional modification values set in advance. Preferably, for the loss weight value and the number of layers, a modification range can be set ; For the number of channels, you can set multiple optional modification values such as 1.2 times, 1.5 times and 1.6 times.

Optionally, when adjusting the model parameters of the initial network model in this step, in addition to adjusting one model parameter at a time, at least two model parameters can also be adjusted at a time, and this step is not limited to this.

S305, if yes, use the trained initial network model as the stylization processing model.

Optionally, if the verification image data is used to verify that the attribute parameters of the initial network model output image after training are greater than the parameter threshold, it means that the initial network model training is completed at this time, and the program code of the initial network model at this time is encapsulated as the style Process the model's program code.

S306, if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image.

S307: Input the target image into the stylized processing model to obtain a preliminary stylized image.

S308: Replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

Embodiments of the present disclosure provide an image stylization processing method. After using sample image data to train the constructed initial network model, if the multi-dimensional stylized processing image output by using the verification image data to verify the output of the trained initial network model is If the attribute parameters are not greater than the parameter threshold, the multi-dimensional model parameters of the initial network model are adjusted and then retrained. Otherwise, the training of the initial network model is completed and the stylized processing model is obtained. In the process of training the initial network model, this embodiment greatly improves the quality of the processing effect of the stylized processing model through the verification of multi-dimensional attribute parameters and the adjustment of multi-dimensional model parameters, and provides a basis for the subsequent stylization of the image to be processed. Processing operations lay the foundation. When stylizing the image to be processed, if the background complexity of the image to be processed is lower than the complexity threshold, the original background of the image to be processed is replaced with the preset template background, and then the stylization processing model is used for stylization. stylization processing operation, and replace the background of the obtained preliminary stylized image with the original background to obtain the final stylized image, taking into account the background complexity of the image to be processed, and further improving the precision and accuracy of the image stylization results. Aesthetics.

Embodiment 3

Figure 4 shows a flow chart of another method for providing video learning materials provided by Embodiment 3 of the present disclosure; this embodiment is optimized based on each option provided by the above embodiment, and specifically provides the following: The target image is input into the stylization processing model to obtain a detailed introduction of the preliminary stylization processing image.

Optionally, as shown in Figure 4, the method in this embodiment may include the following steps:

S401, if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image.

S402: Determine the area to be processed according to the received area selection instruction.

Optionally, in order to improve the user's personalized needs for stylizing the image to be processed, this embodiment may stylize only the user-selected area according to the user's selection. Specifically, after the user selects the image to be processed, the user triggers an area selection instruction through a sliding operation on the image to be processed, where the area selection instruction includes the area to be processed selected by the user. Optionally, if the area selection instruction is triggered by the user's non-closed sliding operation on the image to be processed, the area corresponding to the user's sliding operation can be used as the area to be processed, and the position coordinates corresponding to the area to be processed can be Added to the area selection instruction and transmitted to the electronic device; if the area selection instruction is triggered by the user's closed sliding operation on the image to be processed, the closed frame selection area corresponding to the user's sliding operation can be used as the area to be processed, and The position coordinates corresponding to the area to be processed are added to the area selection command and transmitted to the electronic device. After receiving the area selection instruction triggered by the user, the electronic device obtains the specific coordinate position of the area to be processed from the area selection instruction.

S403. Input the target image and the area to be processed into the stylization processing model, and control the stylization processing model to stylize the content of the area to be processed in the target image to obtain a preliminary stylized image.

Optionally, this step can be to use the target image obtained in S401 and the area to be processed determined in S402 as input data, and call and run the program code of the stylized processing model. At this time, the stylized processing model is based on the input target image and the area to be processed. In the processing area, according to the algorithm during training, only the content of the area to be processed of the target image is stylized. In the obtained preliminary stylized image, only the area to be processed shows the effect of stylization, and the remaining areas are still The effect without stylization.

S404: Replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

Optionally, if the area to be processed determined in S402 does not include a background area, this step may be to replace the background of the initial stylized image obtained in S403 with the original background of the image to be processed to obtain the final stylized image. If the area to be processed determined in S402 includes a background area, this step may be to determine the position coordinates of the area to be processed in the background area, and then use the original background and the position coordinates of the area to be processed in the background area as input data, and input In the background processing model, the background processing model is controlled to only stylize the content of the position coordinate area in the original background to obtain the processed original background; finally, the background of the initial stylized image obtained in S403 is replaced with the processed original background to get the final stylized image.

Embodiments of the present disclosure provide an image stylization processing method. If the user triggers a region selection instruction when selecting an image to be processed, when the background complexity of the image to be processed is lower than the complexity threshold, the background complexity of the image to be processed is changed. The original background is replaced with the preset template background. After obtaining the target image, the area to be processed is determined according to the area selection instruction triggered by the user. The stylized processing model is controlled to stylize only the area to be processed of the target image, and the processed area is processed. The background of the preliminary stylized image is replaced with the original background to obtain the final stylized image. When stylizing the image to be processed, the solution of the embodiment of the present disclosure not only takes into account the complexity of the image background and improves the sophistication and beauty of the image stylization processing results; it can also generate region selection instructions for the user. Personalized stylized processing effects make image stylized processing more interesting.

Embodiment 4

Figure 5 shows a schematic structural diagram of an image stylization processing device provided in Embodiment 4 of the present disclosure. The embodiment of the present disclosure can be applied to the situation of stylizing the image to be processed, especially suitable for background complexity less than complex. The situation where the image to be processed is stylized with a threshold value. The device can be implemented by software and/or hardware, and integrated in the electronic device that executes the method. As shown in Figure 5, the device can include:

The background replacement module 501 is used to, if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, replace the original background of the image to be processed with the template background to obtain the target image;

The stylization processing module 502 is used to input the target image into the stylization processing model to obtain a preliminary stylization processing image;

The background replacement module 501 is also used to replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

Embodiments of the present disclosure provide an image stylization processing device that determines whether the background complexity of the image to be processed is lower than the complexity threshold. If so, the original background of the image to be processed is replaced with a preset template background to obtain the target Image; after the target image is stylized by the stylization processing model, the background of the preliminary stylized image obtained is replaced with the original background to obtain the final stylized image. The solution of the embodiment of the present disclosure takes the complexity of the image background into consideration when stylizing the image. When the background complexity of the image to be processed is low, the method of changing the template background for the image to be processed and then performing stylization processing is adopted. , which avoids the rough processing results caused by the existing stylization processing method for images to be processed with low background complexity, and improves the precision and beauty of the image stylization processing results.

Further, the device further includes a complexity judgment model, which is used when determining that the background complexity of the image to be processed is lower than the complexity threshold:

Texture features of the original background of the image to be processed are extracted, and based on the texture features, it is determined whether the background complexity of the image to be processed is lower than a complexity threshold.

Further, the complexity judgment model is specifically used for:

Perform edge detection on the original background of the image to be processed to obtain texture features of the original background;

Determine the pixel proportion of the texture feature in the original background as the background complexity of the image to be processed;

Determine whether the background complexity of the image to be processed is lower than a complexity threshold.

Further, the device also includes: a model training module, the model training module includes:

The training unit is used to use sample image data to stylize and train the initial network model constructed;

A verification unit, configured to use verification image data to verify whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold; wherein the attribute parameters include: at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter;

A model determination unit configured to use the trained initial network model as the stylization processing model if the attribute parameters of the output image of the trained initial network model are greater than the parameter threshold.

Further, the model training module also includes:

A parameter adjustment unit configured to adjust the model parameters of the initial network model within a preset modification range if the attribute parameters of the trained initial network model output image are less than or equal to the parameter threshold; wherein the model parameters include the initial network At least one of the number of layers, the number of channels, and the loss weight value of the model;

The training unit is also used to use sample image data to re-train the initial network model after adjusting the model parameters in a stylized manner.

Further, the stylization processing module 502 is specifically used to:

Determine the area to be processed according to the received area selection instruction;

The target image and the area to be processed are input into the stylization processing model, and the stylization processing model is controlled to stylize the content of the area to be processed of the target image to obtain a preliminary stylized image.

Further, when the background replacement module 501 replaces the background of the preliminary stylized image with the original background, it is specifically used to:

Input the original background into the background processing model to obtain the processed original background;

Replace the background of the preliminary stylized image with the processed original background.

The image stylization processing device provided by the embodiments of the present disclosure belongs to the same inventive concept as the image stylization processing method provided by the above embodiments. Technical details that are not described in detail in the embodiments of the present disclosure can be referred to the above embodiments, and this disclosure The disclosed embodiments have the same beneficial effects as the above-mentioned embodiments.

Embodiment 5

Referring now to FIG. 6 , a schematic structural diagram of an electronic device 600 suitable for implementing embodiments of the present disclosure is shown. The electronic device may include, but is not limited to, a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Computer), a PMP (Portable Multimedia Player), a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal), etc. mobile terminals such as digital TVs, desktop computers, etc. The electronic device 600 shown in FIG. 6 is only an example and should not bring any limitations to the functions and usage scope of the embodiments of the present disclosure.

As shown in FIG. 6, the electronic device 600 may include a processing device (eg, central processing unit, graphics processor, etc.) 601, which may be loaded into a random access device according to a program stored in a read-only memory (ROM) 602 or from a storage device 608. The program in the memory (RAM) 603 executes various appropriate actions and processes. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, ROM 602 and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 607 such as a computer; a storage device 608 including a magnetic tape, a hard disk, etc.; and a communication device 609. Communication device 609 may allow electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG. 6 illustrates electronic device 600 with various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such embodiments, the computer program may be downloaded and installed from the network via communication device 609, or from storage device 608, or from ROM 602. When the computer program is executed by the processing device 601, the above functions defined in the method of the embodiment of the present disclosure are performed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmed read-only memory (EPROM or flash memory), fiber optics, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, electronic devices may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and may communicate with any form or medium of digital data (e.g., , communication network) interconnection. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or developed in the future network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.

The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the internal process of the electronic device is executed: if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, Then replace the original background of the image to be processed with the template background to obtain the target image; input the target image into the stylized processing model to obtain a preliminary stylized image; replace the background of the preliminary stylized image with The original background is used to obtain the final stylized image.

Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages—such as "C" or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through Internet connection).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.

The units involved in the embodiments of the present disclosure can be implemented in software or hardware. Among them, the name of a unit does not constitute a limitation on the unit itself under certain circumstances.

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

An image stylization processing method is provided according to one or more embodiments of the present disclosure. The method includes:

If it is determined that the background complexity of the image to be processed is lower than the complexity threshold, the original background of the image to be processed is replaced with the template background to obtain the target image;

Input the target image into the stylized processing model to obtain a preliminary stylized image;

The background of the preliminary stylized image is replaced with the original background to obtain a final stylized image.

According to one or more embodiments of the present disclosure, in the above method, determining that the background complexity of the image to be processed is lower than a complexity threshold includes:

Texture features of the original background of the image to be processed are extracted, and based on the texture features, it is determined whether the background complexity of the image to be processed is lower than a complexity threshold.

According to one or more embodiments of the present disclosure, in the above method, extracting texture features of the original background of the image to be processed, and determining whether the background complexity of the image to be processed is lower than a complexity threshold based on the texture features, includes:

Perform edge detection on the original background of the image to be processed to obtain texture features of the original background;

Determine the pixel proportion of the texture feature in the original background as the background complexity of the image to be processed;

Determine whether the background complexity of the image to be processed is lower than a complexity threshold.

According to one or more embodiments of the present disclosure, in the above method, before inputting the target image into the stylization processing model, the method further includes:

Use sample image data to stylize and train the initial network model built;

Verification image data is used to verify whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold; wherein the attribute parameters include: at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter;

If so, the trained initial network model is used as the stylization processing model.

According to one or more embodiments of the present disclosure, in the above method, after using the verification image data to verify whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold, it also includes:

If not, adjust the model parameters of the initial network model within the preset modification range; wherein the model parameters include at least one of the number of layers, the number of channels, and the loss weight value of the initial network model;

The sample image data is used to re-train the initial network model after adjusting the model parameters for stylization processing.

According to one or more embodiments of the present disclosure, in the above method, the target image is input into the stylization processing model to obtain a preliminary stylization processing image, including:

Determine the area to be processed according to the received area selection instruction;

The target image and the area to be processed are input into the stylization processing model, and the stylization processing model is controlled to stylize the content of the area to be processed of the target image to obtain a preliminary stylized image.

According to one or more embodiments of the present disclosure, in the above method, replacing the background of the preliminary stylized image with the original background includes:

Input the original background into the background processing model to obtain the processed original background;

Replace the background of the preliminary stylized image with the processed original background.

An image stylization processing device is provided according to one or more embodiments of the present disclosure. The device includes:

A background replacement module, used to replace the original background of the image to be processed with the template background to obtain the target image if it is determined that the background complexity of the image to be processed is lower than the complexity threshold;

A stylization processing module, used to input the target image into the stylization processing model to obtain a preliminary stylization processing image;

The background replacement module is also used to replace the background of the preliminary stylized image with the original background to obtain the final stylized image.

According to one or more embodiments of the present disclosure, the above-mentioned device further includes a complexity judgment model, which is used when determining that the background complexity of the image to be processed is lower than a complexity threshold:

Texture features of the original background of the image to be processed are extracted, and based on the texture features, it is determined whether the background complexity of the image to be processed is lower than a complexity threshold.

According to one or more embodiments of the present disclosure, the complexity judgment model in the above device is specifically used to:

Perform edge detection on the original background of the image to be processed to obtain texture features of the original background;

Determine the pixel proportion of the texture feature in the original background as the background complexity of the image to be processed;

Determine whether the background complexity of the image to be processed is lower than a complexity threshold.

According to one or more embodiments of the present disclosure, the above device further includes: a model training module, where the model training module includes:

The training unit is used to use sample image data to stylize and train the initial network model constructed;

A verification unit, configured to use verification image data to verify whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold; wherein the attribute parameters include: at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter;

A model determination unit configured to use the trained initial network model as the stylization processing model if the attribute parameters of the output image of the trained initial network model are greater than the parameter threshold.

According to one or more embodiments of the present disclosure, the model training module in the above device further includes:

A parameter adjustment unit configured to adjust the model parameters of the initial network model within a preset modification range if the attribute parameters of the trained initial network model output image are less than or equal to the parameter threshold; wherein the model parameters include the initial network At least one of the number of layers, the number of channels, and the loss weight value of the model;

The training unit is also used to use sample image data to re-train the initial network model after adjusting the model parameters in a stylized manner.

According to one or more embodiments of the present disclosure, the stylization processing module in the above device is specifically used to:

Determine the area to be processed according to the received area selection instruction;

The target image and the area to be processed are input into the stylization processing model, and the stylization processing model is controlled to stylize the content of the area to be processed of the target image to obtain a preliminary stylized image.

According to one or more embodiments of the present disclosure, when the background replacement module in the above device replaces the background of the preliminary stylized image with the original background, it is specifically used to:

Input the original background into the background processing model to obtain the processed original background;

Replace the background of the preliminary stylized image with the processed original background.

An electronic device is provided according to one or more embodiments of the present disclosure. The electronic device includes:

one or more processors;

Memory, used to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the image stylization processing method described in any embodiment of the present disclosure.

A readable medium provided according to one or more embodiments of the present disclosure has a computer program stored thereon, and when the program is executed by a processor, the image stylization processing method as described in any embodiment of the present disclosure is implemented.

The above description is only a description of the preferred embodiments of the present disclosure and the technical principles applied. Those skilled in the art should understand that the disclosure scope involved in the present disclosure is not limited to technical solutions composed of specific combinations of the above technical features, but should also cover solutions composed of the above technical features or without departing from the above disclosed concept. Other technical solutions formed by any combination of equivalent features. For example, a technical solution is formed by replacing the above features with technical features with similar functions disclosed in this disclosure (but not limited to).

Furthermore, although operations are depicted in a specific order, this should not be understood as requiring that these operations be performed in the specific order shown or performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims (8)

1. An image stylization processing method, characterized by including: If it is determined that the background complexity of the image to be processed is lower than the complexity threshold, the original background of the image to be processed is replaced with the template background to obtain the target image, where the background complexity represents the background area in the image to be processed. The complexity threshold is a preset criterion for judging whether to perform background replacement on the image to be processed; Input the target image into the stylized processing model to obtain a preliminary stylized image; Replace the background of the preliminary stylized image with the original background to obtain the final stylized image; Determining that the background complexity of the image to be processed is lower than a complexity threshold includes: Extract texture features of the original background of the image to be processed, and determine whether the background complexity of the image to be processed is lower than a complexity threshold based on the texture features; Extracting texture features of the original background of the image to be processed, and determining whether the background complexity of the image to be processed is lower than a complexity threshold based on the texture features, includes: Perform edge detection on the original background of the image to be processed to obtain texture features of the original background; Determine the pixel proportion of the texture feature in the original background as the background complexity of the image to be processed; Determine whether the background complexity of the image to be processed is lower than a complexity threshold. 2. The method according to claim 1, characterized in that, before inputting the target image into the stylization processing model, it further includes: Use sample image data to stylize and train the initial network model built; Verification image data is used to verify whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold; wherein the attribute parameters include: at least one of a saturation parameter, an edge smoothness parameter, and a shadow parameter; If so, the trained initial network model is used as the stylization processing model. 3. The method according to claim 2, characterized in that, after using the verification image data to verify whether the attribute parameters of the initial network model output image after training are greater than the parameter threshold, it also includes: If not, adjust the model parameters of the initial network model within the preset modification range; wherein the model parameters include at least one of the number of layers, the number of channels, and the loss weight value of the initial network model; The sample image data is used to re-train the initial network model after adjusting the model parameters for stylization processing. 4. The method according to claim 1, characterized in that inputting the target image into a stylized processing model to obtain a preliminary stylized image includes: Determine the area to be processed according to the received area selection instruction; The target image and the area to be processed are input into the stylization processing model, and the stylization processing model is controlled to stylize the content of the area to be processed of the target image to obtain a preliminary stylized image. 5. The method of claim 1, wherein replacing the background of the preliminary stylized image with the original background includes: Input the original background into the background processing model to obtain the processed original background; Replace the background of the preliminary stylized image with the processed original background. 6. An image stylization processing device, characterized in that it includes: A background replacement module, configured to replace the original background of the image to be processed with a template background to obtain the target image if it is determined that the background complexity of the image to be processed is lower than the complexity threshold, where the background complexity represents the The complexity of the background area in the image to be processed. The complexity threshold is a preset criterion for judging whether to perform background replacement on the image to be processed; A stylization processing module, used to input the target image into the stylization processing model to obtain a preliminary stylization processing image; The background replacement module is also used to replace the background of the preliminary stylized image with the original background to obtain the final stylized image; A complexity judgment model, used to extract texture features of the original background of the image to be processed, and determine whether the background complexity of the image to be processed is lower than the complexity threshold based on the texture features; The complexity judgment model is specifically used to perform edge detection on the original background of the image to be processed to obtain the texture features of the original background; Determine the pixel proportion of the texture feature in the original background as the background complexity of the image to be processed; Determine whether the background complexity of the image to be processed is lower than a complexity threshold. 7. An electronic device, characterized in that it includes: one or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the image stylization processing method as described in any one of claims 1-5. 8. A readable medium with a computer program stored thereon, characterized in that when the program is executed by a processor, the image stylization processing method according to any one of claims 1-5 is implemented.