CN116309018A - Image processing method and device - Google Patents

Abstract

The application discloses an image processing method and device thereof, belonging to the technical field of artificial intelligence. The image processing method includes: acquiring an image to be processed; inputting a target image area into a target residual network model to obtain a residual output of the target residual network model; wherein, the target image area includes: the target image area to be processed An image area with hair and forehead is displayed in the image; the residual output is superimposed on the image to be processed to obtain a target image after additional issuance of the image to be processed.

Description

Method and device for image processing

technical field

The present application belongs to the technical field of artificial intelligence, and specifically relates to an image processing method and device thereof.

Background technique

With the continuous growth of age and various factors in social pressure, more and more people gradually start to lose their hair and worry about it. The reduction in hair volume has a great impact on some people. Currently, the StyleGAN (Style Generative Adversarial Network) model can be used to generate renderings after issuance. In this way, even if the user's hair volume is small, it is also possible to generate an additional hair effect map with a large hair volume.

As shown in Figure 1, it is a schematic diagram of the process of using the StyleGAN model to generate the effect map after additional issuance. Among them, the function of Encode (encoder) 11 is to encode the input picture into W+, and W+ is a matrix composed of a series of numbers to represent user information. StyleGAN12 achieves additional issuance by decoding W+ information and adjusting hair information.

However, the above-mentioned use of the StyleGAN model for additional issuance has high performance requirements for the hardware part, and the power consumption overhead is relatively large.

Contents of the invention

The purpose of the embodiments of the present application is to provide an image processing method and device thereof, which can solve the problems in the related art that the performance of the hardware part is high and the power consumption is relatively high when realizing the effect of additional issuance of characters.

In the first aspect, the embodiment of the present application provides an image processing method, the method comprising:

Get the image to be processed;

The target image area is input to the target residual network model to obtain the residual output of the target residual network model; wherein, the target image area includes: an image area showing hair and forehead in the image to be processed;

The residual output is superimposed on the image to be processed to obtain a target image of the image to be processed.

In the second aspect, the embodiment of the present application provides an image processing device, which includes:

The first acquisition module is used to acquire the image to be processed;

The residual module is used to input the target image area into the target residual network model to obtain the residual output of the target residual network model; wherein, the target image area includes: the image to be processed is displayed with hair and the image area of the forehead;

A processing module, configured to superimpose the residual output on the image to be processed to obtain a target image after the image to be processed is added.

In the third aspect, the embodiment of the present application provides an electronic device, the electronic device includes a processor and a memory, the memory stores programs or instructions that can run on the processor, and the programs or instructions are processed by the The steps of the method described in the first aspect are realized when the controller is executed.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or an instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented .

In the fifth aspect, the embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions, so as to implement the first aspect the method described.

In a sixth aspect, an embodiment of the present application provides a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the method in the first aspect.

In the embodiment of the present application, for the image to be processed, the target image area displaying hair and forehead is input to the target residual network model, and the residual output representing the difference before and after the image to be processed can be obtained. Furthermore, by superimposing the residual output on the image to be processed, the target image after additional issuance of the image to be processed can be obtained. Since the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption overhead. Therefore, the present application can reduce the performance requirements of the hardware and reduce the power consumption of the hardware on the basis of realizing the effect of additional issuance.

Description of drawings

Figure 1 is a schematic diagram of the current process of using the StyleGAN model to generate the effect map after additional issuance;

FIG. 2 is a flow chart of the steps of an image processing method provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of changes before and after additional issuance of images to be processed provided in the embodiment of the present application;

Fig. 4 is a schematic diagram of the training style generation confrontation network model provided by the embodiment of the present application;

Fig. 5 is a schematic diagram of obtaining a pair of comparison images before and after additional issuance provided by the embodiment of the present application;

FIG. 6 is a schematic diagram of a practical application of the image processing method provided by the embodiment of the present application;

FIG. 7 is a structural block diagram of an image processing device provided in an embodiment of the present application;

FIG. 8 is one of the schematic diagrams of the hardware structure of the electronic device provided by the embodiment of the present application;

FIG. 9 is a second schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present application.

Detailed ways

The following will clearly describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in this application belong to the protection scope of this application.

The terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific sequence or sequence. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in sequences other than those illustrated or described herein and that references to "first", "second", etc. to distinguish Objects are generally of one type, and the number of objects is not limited. For example, there may be one or more first objects. In addition, "and/or" in the specification and claims means at least one of the connected objects, and the character "/" generally means that the related objects are an "or" relationship.

The image processing method provided by the embodiment of the present application will be described in detail below through specific embodiments and application scenarios with reference to the accompanying drawings.

As shown in FIG. 2, it is a schematic diagram of the steps of an image processing method provided in the embodiment of the present application. The image processing method may include:

Step 201: Obtain an image to be processed.

In this step, the image to be processed is the image to which additional issuance effects need to be added, that is, the image before additional issuance. It can be an image captured by electronic devices such as mobile phones and cameras. Here, the image to be processed may be obtained locally. For example, the method provided in this application is applied to a mobile phone, and the image to be processed may be a local image stored in the mobile phone. Of course, images to be processed transmitted from other electronic devices may also be received. For example, the method provided in this application is applied to a server, and the image to be processed may be an image uploaded to the server by a mobile phone user.

It can be understood that the image content of the image to be processed includes at least hair and forehead. Specifically, the image to be processed includes an image showing a hairline. It is worth noting that the size of the image to be processed is not limited here. That is to say, the size of the image to be processed can be any size. For example, the size of the image to be processed may be 1536*768, 4096*4096, but not limited thereto.

Step 202: Input the target image region into the target residual network model to obtain a residual output of the target residual network model.

It should be noted that the target image area includes: an image area displaying hair and forehead in the image to be processed. Specifically, the target image area is all or part of the image area of the image to be processed. In the case that the target image area is a partial image area of the image to be processed, the partial image area is an image area displaying hair and forehead.

The target residual network model is used to calculate the difference before and after the additional issuance of characters. Specifically, the target residual network model is a model based on a residual network. The target residual network model can calculate the difference before and after the additional issuance of characters, and output the difference as a residual. Among them, the additional hair of a character refers to the increase of the hair or hair volume of a character. The amount of hair after the additional issuance of characters is significantly higher than that before the additional issuance.

Step 203: Superimpose the residual output on the image to be processed to obtain the target image after the image to be processed is added.

It should be noted that the residual output is the difference before and after the additional issuance of characters. Here, the image to be processed is taken as the image before the person is added, so that the image after the person is added can be obtained by superimposing the residual output with the image before the person is added, that is, the target image after the image to be processed is added. It can be understood that the location of additional issuance may be the location of the hairline of the image to be processed. Therefore, compared with the image to be processed, the target image looks like the hairline of the character has moved toward the forehead. As shown in FIG. 3 , the target image 33 can be obtained after superimposing the image to be processed 31 and the residual output 32 . Obviously, compared with the image to be processed 31, the hairline in the target image 33 is closer to the face, so that the exposed forehead area is smaller. Of course, the location of additional issuance may also be a location where the hair is thinner in the image to be processed, so that compared with the image to be processed, the part of the target image that looks like a character with thin hair becomes denser.

In the embodiment of the present application, for the image to be processed, the target image area displaying hair and forehead is input to the target residual network model, and the residual output representing the difference before and after the image to be processed can be obtained. Furthermore, by superimposing the residual output on the image to be processed, the target image after additional issuance of the image to be processed can be obtained. Since the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption overhead. Therefore, the present application can reduce the performance requirements of the hardware and reduce the power consumption of the hardware on the basis of realizing the effect of additional issuance.

Optionally, before inputting the target image region into the target residual network model to obtain the residual output of the target residual network model, the method further includes:

Acquiring multiple pairs of comparison images before and after issuance; wherein, each pair of comparison images includes an image before issuance and an image after issuance.

Taking the image before additional issuance as input and the image after additional issuance as the real value, the initial residual network model is trained to obtain the target residual network model.

It should be noted that the initial residual network model is an untrained model based on the residual network. Among them, the residual network can be built in the network form of encoding + decoding structure, for example, the residual network can be built using UNet. In the process of training the initial residual network model, the learning rate and loss function can be set according to experience. For example, the learning rate can be set to 0.001, and the loss function can adopt a combination of LPIPS (Learned Perceptual Image Patch Similarity) loss function + L1 loss function, but not limited thereto. Among the multiple pairs of comparison images before and after additional issuance, each pair of comparison images before and after additional issuance is an image of the same person, that is, the image of the person before the additional issuance and the image of the person after the additional issuance, and the image of the person after the additional issuance is the image before the additional issuance Based on the obtained image, increase the hair volume. When obtaining multiple pairs of comparison images before and after additional issuance, you can first obtain a certain number of person images, and then draw a person image after additional issuance for each person image, or use the current application that can achieve the effect of additional issuance to generate additional person images , the image of the person and the image of the person after additional issuance are used as a pair of comparison images before and after additional issuance, but it is not limited thereto.

Multiple pairs of comparison images before and after additional issuance are used to perform model training on the initial residual network model, so as to obtain a trained model, that is, the target residual network model. Here, there is no limit to the number of comparison images before and after additional issuance. Specifically, each pair of comparison images before and after additional issuance can be used as a sample for model training. In the way of supervised learning, the initial residual network model is trained by using multiple pairs of comparison images before and after additional issuance until the model converges or the number of training times reaches the threshold. It is worth noting that for the supervised learning of the residual network, in the training process of each model, the pre-increment image in a pair of comparison images before and after the increase is used as the input (INPUT), and the image after the increase is used as the real value (ground truth, referred to as GT), expand the model training. Through continuous model training, the residual output in the model can gradually and accurately represent the difference before and after the additional issuance of characters.

In the embodiment of the present application, multiple pairs of comparison images before and after additional issuance are used as training samples, and then the training samples are used to train the initial residual network model, so that the target residual network model for calculating the difference before and after additional issuance of characters can be obtained.

Optionally, obtain multiple pairs of comparison images before and after additional issuance, including:

Get multiple 3-D matrices that fit a Gaussian distribution.

For each three-dimensional matrix, the adversarial network model is generated based on the pre-trained style, and a pair of comparison images before and after additional issuance are generated.

It should be noted that the model input of the trained style generation confrontational network model is a plurality of three-dimensional matrices conforming to the Gaussian distribution, and the model output is an image containing hair and forehead. That is to say, any three-dimensional matrix conforming to the Gaussian distribution is input into the trained style generation confrontation network model, and an image containing hair and forehead can be obtained. Finally, using the model output, a pair of images containing hair and forehead can be obtained as a pair of comparison images before and after additional issuance. Preferably, the three-dimensional matrix acquired here is a randomly generated three-dimensional matrix.

In the embodiment of the present application, based on the three-dimensional matrix and the pre-trained style generation confrontation network model, it is possible to quickly generate comparison images before and after additional issuance, and at the same time simplify the production process of comparison images before and after additional issuance.

Optionally, before each three-dimensional matrix generates a confrontational network model based on a pre-trained style, and generates a pair of comparison images before and after additional issuance, the method further includes:

Obtain a training sample set; wherein, the training sample set includes a plurality of random three-dimensional matrices conforming to Gaussian distribution, training images including forehead and hair;

Based on the training sample set, the target model is trained to obtain a pre-trained style generation confrontation network model;

Among them, the target model includes: an initial style generation confrontation network model and a discriminator model, and the output of the initial style generation confrontation network model and the training image are used as the input of the discriminator model.

It should be noted that the training sample set is used for training the target model. Thanks to the trained style GAN model can generate images containing hair and forehead. Therefore, training images containing foreheads and hair need to be used. There is no limit to the size of the training images, but it is required that each training image has the same size. Specifically, when obtaining training images, first obtain high-definition portrait data captured by a single-lens reflex camera; then perform face detection and face rotation correction alignment on the high-definition portrait data. Finally, crop the hair area of the rotated and aligned face, and uniformly crop it to a size of 2:1 width*height (other sizes are also available, here we take 1536*768 as an example), and the cropped image is the training image.

Since the target model includes: the initial style generation confrontation network model and the discriminator model, and the output of the initial style generation confrontation network model and the training image are used as the input of the discriminator model. Therefore, the process of training the target model is the process of training the initial style generative adversarial network model in the target model, and the initial style generative adversarial network model after the training is the trained style generative adversarial network model. Among them, the initial style generation confrontation network model is used as a generator, which can generate an image based on a random three-dimensional matrix conforming to the Gaussian distribution. Therefore, the initial style generation adversarial network model here can also be understood as the synthesis network part of the style generation adversarial network model composed of a mapping network (Mapping network) and a synthesis network (Synthesis network). It is understandable that during each training process of the target model, a random three-dimensional matrix needs to be input to the initial style generation confrontational network model, and then the model output of the initial style generation confrontational network model and a training image are input into the discriminator model , to get the model output of the discriminator model. After calculating the model loss, the model parameters of the initial style generative adversarial network model and the discriminator model are reversely updated using the backpropagation algorithm. After the initial style generative adversarial network model has fully converged, the model training is stopped.

Specifically, as shown in FIG. 4 , a generator 41 may be initialized first, and a discriminator model 42 may be initialized.

First execute step 401: randomly generate a three-dimensional matrix W+ conforming to the Gaussian distribution, and input W+ into the generator 41 to obtain an RGB image 43 output by the generator 41, wherein RGB represents the colors of three channels of red, green and blue. Now use a training image 44 as a real hair picture, and set its corresponding label as True (correct). Use the RGB image 43 as a fake hair picture, and set its corresponding label as False (wrong), at this time, use the loss function formula 1 to calculate the loss, and backpropagate to update the model parameters of the discriminator model;

loss=log(exp(D(G(W+)))+1)+log(exp(-D(x))+1) (1)

Among them, G(W+) represents RGB image 43 , x training image 44 , and D represents discriminator model 42 .

Execute step 402: re-randomly generate a three-dimensional matrix W+' conforming to the Gaussian distribution, and input W+' into the generator 41 to obtain a new RGB image. At this time, there is a trained discriminator model 42 and a new RGB image. At this time, the loss is calculated by the loss function formula 2, and the weight of the generator 41 is updated through backpropagation;

loss=-log(exp(D(G(W+)))+1) (2)

By repeatedly executing these two steps in the order of step 401 and step 402, the generator 41 can be trained until the generator 41 converges completely. At this point, a mapping relationship from W+ to 1536*768 images will be obtained. Any random three-dimensional matrix conforming to the Gaussian distribution is input to the generator 41 and a corresponding 1536*768 image including hair and forehead will be obtained.

In the embodiment of the present application, the training of the style generation confrontation network model can be realized by using the random three-dimensional matrix and the training image containing the forehead and hair, so as to obtain the network model used to generate the image containing the forehead and hair using the three-dimensional matrix.

Optionally, for each three-dimensional matrix, generate an adversarial network model based on the pre-trained style respectively, and generate a pair of comparison images before and after additional issuance, including:

Multiple three-dimensional matrices are respectively input into the style generation confrontation network model, and the pre-issued images output by the style generation confrontation network model are respectively obtained.

Based on the output of the style generation confrontation network model and the guide image, the input of the style generation confrontation network model is reversely adjusted until the number of reverse adjustments reaches the target number of times; wherein, the guide image is a comparison image after issuance based on the image before issuance .

After the last reverse adjustment, the output of the style generation confrontational network model is determined as the additional image.

It should be noted that a pair of comparison images before and after additional issuance can be generated for each three-dimensional matrix, and this embodiment of the present application only uses a three-dimensional matrix to generate a pair of comparison images before and after additional issuance as an example. Since any random three-dimensional matrix conforming to the Gaussian distribution is input to the style generation confrontation network model, a corresponding image containing hair and forehead will be obtained. Therefore, the image obtained for the first time is taken as the pre-issue image in a pair of comparison images before and after additional issuance. Then, based on the image before the additional issuance, a comparison image after the additional issuance is drawn, and the comparison image after the additional issuance is used as a guide image to reversely adjust the three-dimensional matrix. When the number of reverse adjustments reaches the target number, the image output by the style generation confrontational network model will be very close to the guide image, so the output image at this time can be used as the post-issue image in a pair of comparison images before and after the increase.

As shown in Figure 5, in order to obtain a flow chart of the pre-issue image 51 and the image 52 after the additional issue in a pair of comparison images before and after the additional issue, specifically, it includes:

Step 501: Input a three-dimensional matrix W+ conforming to the Gaussian distribution into the style generation confrontational network model 53 to obtain the pre-issue image 51, and store the pre-issue image 51 as the pre-issue image in a pair of comparison images before and after the increase.

Step 502: Use a paintbrush to draw the hair of the hair growth area in the image 51 before the hair growth (only the hair color can be drawn), and generate the guide image 54 .

Step 503: Use the L1 loss function to calculate the difference between the pre-issued image 51 and the guide image 54, and obtain the effect gap loss=|before-issued image 51−guided image 54| of the two images.

Step 504: The learning rate in the model training process is set to 0.01, and the optimizer is set to Adam (Adaptive Moment Estimation, Adaptive Moment Estimation) optimizer, using the backpropagation method to generate an adversarial network model 53 along the style of loss The backpropagation to the three-dimensional matrix W+, at this time, the gradient in the backpropagation is used to update the numbers in the three-dimensional matrix W+ to obtain a new three-dimensional matrix W+*.

Step 505: Input the new three-dimensional matrix W+* into the style generation confrontational network model 53 to obtain a new image before additional issuance.

Step 506: Use the L1 loss function to calculate the difference between the new pre-issue image and the guide image 54, and obtain the effect gap loss=|new pre-issue image-guidance image 54| of the two images.

Steps 504 to 506 are repeatedly executed for a target number of times. Among them, in the process of repeated execution, the style generation confrontation network model 53 will refer to the guide image 54 to gradually grow the hair on the image 51 before additional issuance, and fill in the hair texture. The target number of times can be set arbitrarily, for example, the target number of times can be 100 times, but it is not limited thereto. The image finally output by the style generation confrontational network model 53 is used as the post-issue image in a pair of comparison images before and after additional issue.

In the embodiment of the present application, after the image before additional issuance is generated, the drawn comparison image after additional issuance is used as the guiding image. By continuously adjusting the input of the style generation confrontational network model in reverse, the image after the increase can be obtained, and the size of the image before and after the increase is the same.

Optionally, the input of the target residual network model is an image of the first size, and when the size of the image to be processed is smaller or larger than the first size, the target image area is input to the target residual network model to obtain the target residual The residual output of the network model, including:

Affine transformation is performed on the target image area to generate a first intermediate image of the first size;

The first intermediate image is input to the target residual network model to obtain the residual output of the target residual network model;

Superimpose the residual output on the image to be processed to obtain the target image after the additional image to be processed, including:

Performing an inverse affine transformation on the residual output to generate a second intermediate image of a second size, where the second size is the size of the image to be processed;

The second intermediate image is superimposed on the image to be processed to obtain the target image.

It should be noted that the target residual network model is a pre-trained network model, and the size of its input image will be fixed after the training. Since the size of the image to be processed does not necessarily meet the size requirements of the target residual network model for the input image. Therefore, the size of the image to be processed can be detected in advance. If its size is the first size, that is, the size of the image to be processed meets the size requirement of the target residual network model for the input image, then the image to be processed can be used directly. Otherwise, affine transformation is required, and an inverse affine transformation is performed on the residual output, so that the size of the residual output is consistent with the size of the image to be processed, and various problems caused by inconsistent sizes are avoided. For example, use 1024 resolution image training to get the target residual network model. When the original image (image to be processed) is 4096 resolution, use affine transformation to downsample to 1024 resolution, then feed the target residual network model to generate 1024 resolution residual output, and then enlarge the residual output Up to 4096 resolution superimposed on the original image.

In the embodiment of the present application, by performing affine transformation on the image to be processed and performing inverse affine transformation on the residual output, images to be processed of various sizes can be processed without affecting the definition of the target image.

Optionally, affine transformation is performed on the target image area to generate a first intermediate image of the first size, including:

Crop the target image area in the image to be processed to obtain a cropped image.

Affine transformation is performed on the cropped image to generate a first intermediate image of the first size.

It should be noted that, since the method provided by the present application is to achieve the effect of additional issuance of the image to be processed. Therefore, only the target image area in the image to be processed can be processed. Here, the target image area is cropped to obtain a cropped image.

In the embodiment of the present application, the cropped image can be obtained by cropping the target image area in the image to be processed. Furthermore, the cropped image is used to obtain the residual output, which can avoid the influence caused by the information of other image areas in the image to be processed except the target image area.

As shown in Figure 6, it is a schematic diagram of the practical application of the image processing method provided by the embodiment of this application; this application regards the resolution as the size of the image, and generates an additional target for the image 61 to be processed with a size of 4096*4096 Image 62. Assume that the size of the input image of the trained target residual network model 63 is 1536*768. The process then includes:

Step 601: Obtain an image 61 to be processed with a size of 4096*4096.

Step 602 : For the image to be processed 61 , detect the frame of the human face and crop the hair and forehead image areas to obtain a cropped image 64 .

Step 603: Scale the cropped image 64 to 1536*768 to obtain the original intermediate image 65.

Step 604: Input the original intermediate image 65 of 1536*768 into the target residual network model 63 to obtain a residual output 66 of 1536*768. Wherein, the residual output 66 can be understood as the residual corresponding to the original intermediate image 65 .

Step 605: Scale the residual output 66 of 1536*768 to an intermediate residual output 67 of 4096*4096. Wherein, the intermediate residual output 67 can be understood as corresponding to the cropped image 64 .

Step 606 : Based on the intermediate residual output 67 , generate a residual corresponding to the image to be processed 61 , that is, a target residual 68 . The target residual 68 is superimposed on the clipped area of the image 61 to be processed to obtain the final rendering, namely the target image 62 .

In the embodiment of the present application, by designing the residual network, it is possible to improve the definition of the final picture while controlling the performance and power consumption, and solve the problem of the reduction of definition, excessive performance and power consumption, and Operators do not support the disadvantages.

It should be noted that, the image processing method provided in the embodiment of the present application may be executed by an image processing device, or a control module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking an image processing device performing an image processing method as an example.

As shown in Figure 7, the embodiment of the present application also provides an image processing device, which includes:

A first acquisition module 71, configured to acquire an image to be processed;

The residual module 72 is used to input the target image area to the target residual network model to obtain the residual output of the target residual network model; wherein, the target image area includes: an image area showing hair and forehead in the image to be processed;

The processing module 73 is configured to superimpose the residual output on the image to be processed to obtain a target image after the image to be processed is added.

Optionally, the device also includes:

The second acquisition module is used to acquire multiple pairs of comparison images before and after issuance; wherein, each pair of comparison images includes an image before issuance and an image after issuance;

The first training module is used to train the initial residual network model by using the pre-issued image as input and the post-issued image as a real value to obtain a target residual network model.

Optionally, the second acquisition module includes:

An acquisition unit, configured to acquire a plurality of three-dimensional matrices conforming to Gaussian distribution;

The generation unit is configured to generate a confrontational network model based on a pre-trained style for each three-dimensional matrix, and generate a pair of comparison images before and after additional issuance.

Optionally, the device also includes:

The third acquisition module is used to acquire a training sample set; wherein, the training sample set includes a plurality of random three-dimensional matrices conforming to Gaussian distribution, training images including forehead and hair;

The second training module is used to train the target model based on the training sample set to obtain a pre-trained style generation confrontation network model;

Among them, the target model includes: an initial style generation confrontation network model and a discriminator model, and the output of the initial style generation confrontation network model and the training image are used as the input of the discriminator model.

Optionally, generate units, specifically for:

Input multiple three-dimensional matrices into the style generation confrontation network model respectively, and respectively obtain the pre-issued images output by the style generation confrontation network model;

Based on the output of the style generation confrontation network model and the guide image, the input of the style generation confrontation network model is reversely adjusted until the number of reverse adjustments reaches the target number of times; wherein, the guide image is a comparison image after issuance based on the image before issuance ;

After the last reverse adjustment, the output of the style generation confrontational network model is determined as the additional image.

Optionally, the input of the target residual network model is an image of the first size, and when the size of the image to be processed is smaller or larger than the first size, the residual module 72 includes:

An affine transformation unit, configured to perform affine transformation on the target image area to generate a first intermediate image of the first size;

The residual unit is used to input the first intermediate image to the target residual network model to obtain the residual output of the target residual network model;

Processing module 73, comprising:

An inverse affine transformation unit, configured to perform inverse affine transformation on the residual output to generate a second intermediate image of a second size, where the second size is the size of the image to be processed;

The processing unit is configured to superimpose the second intermediate image on the image to be processed to obtain the target image.

Optionally, an affine transformation unit, specifically for:

Crop the target image area in the image to be processed to obtain a cropped image;

Affine transformation is performed on the cropped image to generate a first intermediate image of the first size.

In the embodiment of the present application, for the image to be processed, the target image area displaying hair and forehead is input to the target residual network model. Since the target residual network model is used to calculate the difference before and after the person is added, the residual output representing the difference before and after the image to be processed can be obtained. Furthermore, by superimposing the residual output on the image to be processed, the target image after additional issuance of the image to be processed can be obtained. Since the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption overhead. Therefore, the present application can reduce the performance requirements of the hardware and reduce the power consumption of the hardware on the basis of realizing the effect of additional issuance.

The image processing apparatus in the embodiment of the present application may be an electronic device, or may be a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or other devices other than the terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR ) equipment, robots, wearable devices, ultra-mobile personal computer (ultra-mobilepersonal computer, UMPC), netbook or personal digital assistant (personal digital assistant, PDA), etc. ), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which are not specifically limited in this embodiment of the present application.

The image processing device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present application.

The image processing device provided by the embodiment of the present application can implement the various processes realized by the method embodiments in Fig. 2-Fig. 6 and achieve the same technical effect. In order to avoid repetition, details are not repeated here.

Optionally, as shown in FIG. 8 , the embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, and the memory 802 stores programs or instructions that can run on the processor 801. The When the programs or instructions are executed by the processor 801, the various steps of the above image processing method embodiments can be realized, and the same technical effect can be achieved, so in order to avoid repetition, details are not repeated here.

It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

FIG. 9 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 900 includes, but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910, etc. part.

Those skilled in the art can understand that the electronic device 900 can also include a power supply (such as a battery) for supplying power to various components, and the power supply can be logically connected to the processor 910 through the power management system, so that the management of charging, discharging, and function can be realized through the power management system. Consumption management and other functions. The structure of the electronic device shown in FIG. 9 does not constitute a limitation to the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine some components, or arrange different components, which will not be repeated here. .

Processor 910, configured to acquire images to be processed.

The processor 910 is further configured to input the target image area into the target residual network model to obtain the residual output of the target residual network model; wherein, the target image area includes: an image area showing hair and forehead in the image to be processed.

The processor 910 is further configured to superimpose the residual output on the image to be processed to obtain a target image after the image to be processed is added.

In the embodiment of the present application, for the image to be processed, the target image area displaying hair and forehead is input to the target residual network model, and the residual output representing the difference before and after the image to be processed can be obtained. Furthermore, by superimposing the residual output on the image to be processed, the target image after additional issuance of the image to be processed can be obtained. Since the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption overhead. Therefore, the present application can reduce the performance requirements of the hardware and reduce the power consumption of the hardware on the basis of realizing the effect of additional issuance.

It should be understood that, in this embodiment of the present application, the input unit 904 may include a graphics processing unit (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processing unit 9041 is compatible with an image capture device (such as Camera) to process the image data of still pictures or videos. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072 . The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts, a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.

The memory 909 can be used to store software programs as well as various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required by at least one function (such as a sound playing function, image playback function, etc.), etc. Furthermore, memory 909 may include volatile memory or nonvolatile memory, or memory x09 may include both volatile and nonvolatile memory. Wherein, the non-volatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electronically programmable Erase Programmable Read-Only Memory (Electrically EPROM, EEPROM) or Flash. Volatile memory can be random access memory (Random Access Memory, RAM), static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (Synch link DRAM , SLDRAM) and Direct Memory Bus Random Access Memory (Direct Rambus RAM, DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and application programs, etc., Modem processors mainly process wireless communication signals, such as baseband processors. It can be understood that the foregoing modem processor may not be integrated into the processor 910 .

The embodiment of the present application also provides a readable storage medium, the readable storage medium stores a program or an instruction, and when the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is realized, and can achieve The same technical effects are not repeated here to avoid repetition.

Wherein, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

The embodiment of the present application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, the processor is used to run programs or instructions, and implement the above image processing method embodiment Each process, and can achieve the same technical effect, in order to avoid repetition, will not repeat them here.

It should be understood that the chips mentioned in the embodiments of the present application may also be called system-on-chip, system-on-chip, system-on-a-chip, or system-on-a-chip.

An embodiment of the present application provides a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the various processes in the above image processing method embodiment, and can achieve the same technology Effect, in order to avoid repetition, will not repeat them here.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions are performed, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present application can be embodied in the form of computer software products, which are stored in a storage medium (such as ROM/RAM, magnetic disk, etc.) , optical disc), including several instructions to enable a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

Claims (10)

1. A method for image processing, characterized in that the method comprises: Get the image to be processed; The target image area is input to the target residual network model to obtain the residual output of the target residual network model; wherein, the target image area includes: an image area showing hair and forehead in the image to be processed; The residual output is superimposed on the image to be processed to obtain a target image of the image to be processed. 2. The method according to claim 1, characterized in that, before the target image region is input to the target residual network model to obtain the residual output of the target residual network model, the method further comprises: Obtaining multiple pairs of comparison images before and after issuance; wherein, each pair of comparison images includes an image before issuance and an image after issuance; Using the pre-issued image as an input and the post-issued image as a real value, train an initial residual network model to obtain the target residual network model. 3. The method according to claim 2, wherein said acquiring multiple pairs of comparison images before and after additional issuance comprises: Obtain multiple three-dimensional matrices conforming to Gaussian distribution; For each of the three-dimensional matrices, an adversarial network model is generated based on a pre-trained style, and a pair of comparison images before and after additional issuance are generated. 4. The method according to claim 3, wherein, for each of the three-dimensional matrices, generating an adversarial network model based on a pre-trained style respectively, generating a pair of comparison images before and after additional issuance, including: Inputting the plurality of three-dimensional matrices into the style generation confrontational network model respectively, and respectively obtaining the pre-issued images output by the style generation confrontational network model; Based on the output of the style generation confrontation network model and the guide image, the input of the style generation confrontation network model is reversely adjusted until the number of reverse adjustments reaches the target number of times; wherein, the guide image is based on the pre-issued The comparison image after the additional issuance of the image drawing; After the last reverse adjustment, the output of the style generation adversarial network model is determined as the additional image. 5. The method according to claim 1, wherein the input of the target residual network model is an image of a first size, and when the size of the image to be processed is smaller or larger than the first size , the target image area is input to the target residual network model, and the residual output of the target residual network model is obtained, including: performing affine transformation on the target image area to generate a first intermediate image of a first size; Inputting the first intermediate image into the target residual network model to obtain a residual output of the target residual network model; The superimposing the residual output on the image to be processed to obtain the target image after additional issuance of the image to be processed includes: performing an inverse affine transformation on the residual output to generate a second intermediate image of a second size, wherein the second size is the size of the image to be processed; Superimposing the second intermediate image on the image to be processed to obtain the target image. 6. A device for image processing, characterized in that the device comprises: The first acquisition module is used to acquire the image to be processed; The residual module is used to input the target image area into the target residual network model to obtain the residual output of the target residual network model; wherein, the target image area includes: the image to be processed is displayed with hair and the image area of the forehead; A processing module, configured to superimpose the residual output on the image to be processed to obtain a target image after the image to be processed is added. 7. The device according to claim 6, further comprising: The second acquisition module is used to acquire multiple pairs of comparison images before and after issuance; wherein, each pair of comparison images includes an image before issuance and an image after issuance; The first training module is configured to use the image before issuance as input and the image after issuance as a real value to train an initial residual network model to obtain the target residual network model. 8. The device according to claim 7, wherein the second acquiring module comprises: An acquisition unit, configured to acquire a plurality of three-dimensional matrices conforming to Gaussian distribution; The generating unit is configured to generate a confrontational network model based on a pre-trained style for each of the three-dimensional matrices, and generate a pair of comparison images before and after additional issuance. 9. The device according to claim 8, wherein the generating unit is specifically used for: Inputting the plurality of three-dimensional matrices into the style generation confrontational network model respectively, and respectively obtaining the pre-issued images output by the style generation confrontational network model; Based on the output of the style generation confrontation network model and the guide image, the input of the style generation confrontation network model is reversely adjusted until the number of reverse adjustments reaches the target number of times; wherein, the guide image is based on the pre-issued The comparison image after the additional issuance of the image drawing; After the last reverse adjustment, the output of the style generation adversarial network model is determined as the additional image. 10. The device according to claim 6, wherein the input of the target residual network model is an image of a first size, and when the size of the image to be processed is smaller or larger than the first size , the residual module includes: an affine transformation unit, configured to perform affine transformation on the target image area to generate a first intermediate image of a first size; a residual unit, configured to input the first intermediate image into the target residual network model, and obtain a residual output of the target residual network model; The processing module includes: an inverse affine transformation unit, configured to perform inverse affine transformation on the residual output to generate a second intermediate image of a second size, wherein the second size is the size of the image to be processed; A processing unit, configured to superimpose the second intermediate image on the image to be processed to obtain the target image.

Images