CN114418872A - A real image aesthetic enhancement method based on mGANprior - Google Patents

Abstract

The invention discloses a real image aesthetic feeling enhancement method based on mGANPrior, for a real image to be enhanced in aesthetic feeling, selecting a PGGAN pre-training generation model of a corresponding type, and determining the type of an aesthetic feeling effect needing to be enhanced; performing semantic segmentation on the real image by using a cascade segmentation module method; obtaining an inverse mapping image by using an mGANNprior method; according to the aesthetic style to be enhanced, corresponding degradation transformation is carried out on the real image and the inversely mapped image, loss is calculated, and a final hidden vector and an image I with the enhanced image aesthetic style are obtained through gradient descent optimization_enh(ii) a The method of the invention realizes the aesthetic enhancement of the real image, not only retains the original information of the image to the maximum extent, but also aligns the imageLike controllable aesthetic style modification, the invention provides a loss function of degradation transformation according to aesthetic factors, and can generate aesthetic fuzzy effect on a real image.

Description

A real image aesthetic enhancement method based on mGANprior

technical field

The invention relates to the field of inverse mapping of generative adversarial networks, semantic segmentation of images and image aesthetic enhancement, in particular to inverse mapping of real images to latent spaces of generative adversarial networks, and constructing aesthetic degradation transformations for two aesthetic styles for loss calculation, thereby To achieve the effect of enhancing the beauty of real images.

Background technique

The research based on Generative Adversarial Network (GAN) has made great breakthroughs and progress in recent years. Scholars have developed a large number of high-quality GAN-based derivative models such as PGGAN, StyleGAN, BigGAN, etc. PGGAN is the first generative model that proposes a progressive training method to obtain high-quality generated images. The model can finally obtain high-quality images of 1024×1024 through the progressive training method. PGGAN provides pre-trained generative models for multiple scenarios, such as churches, towers, bridges, bedrooms, etc.

The GAN model maps the latent space to the image space, and the GAN inverse mapping is its inverse process, that is, the mapping from the image space to the latent space is established. GAN inverse mapping aims to map the real image back into the latent space of the pre-trained GAN model, which can then be reconstructed from the latent space obtained by the inverse mapping by the generative model. mGANprior is a GAN inverse mapping method that can reconstruct real images with high quality. It has been verified that this method has excellent inverse mapping effects on various indoor and outdoor scene generation models. At the same time, before calculating the loss between the generated image and the real image, mGANprior realizes image processing such as coloring, super-resolution reconstruction and image restoration of the real image by applying different degradation transformations to the image.

Semantic segmentation is a deep learning algorithm that associates a label or category with each pixel of an image. It is used to identify collections of pixels that constitute distinguishable categories. For example, a bridge landscape photo needs to identify bridges, creeks, grass, trees, sky, etc. The Cascade Segmentation Module method is a general semantic segmentation method proposed in the paper "Scene Parsing through ADE20K Dataset" (Scene Parsing through ADE20K Dataset), which parses the scene into materials, objects and object parts in a cascade manner. , which can be applied to semantic segmentation in different scenarios.

Based on the above background, if the degradation transformation method is designed for 14 types of aesthetic styles, the real image aesthetic enhancement based on GAN inverse mapping can be realized. The premise is that the degradation transformation is known and derivable. For example, the degradation transformations corresponding to image colorization, super-resolution reconstruction, and image restoration are grayscale, downsampling, and image cropping, which are all known and derivable. . However, there is still no derivable transformation method for aesthetic factors based on the current research results. Therefore, the present invention intends to design a degradation transformation method for aesthetic style, so as to realize the enhancement of real image aesthetics based on GAN.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention proposes a real image beauty enhancement method based on mGANprior, and designs different degradation transformation methods for the two fuzzy aesthetic styles of shallow depth of field and motion blur, so as to realize the enhancement of the fuzzy beauty of real images. The technical solution includes the following steps:

Step 1: Select a real image I to be enhanced, select the corresponding type of PGGAN pre-training generation model, and determine whether the image needs to be enhanced with shallow depth of field or motion blur.

Step 2: Use the Cascade Segmentation Module method to perform semantic segmentation on the real image I, and extract the main pixels. And according to the need to enhance the aesthetic effect (shallow depth of field or motion blur), different binary matrices m are obtained.

Step 3: Use the mGANprior method to obtain the inversely mapped image I _inv .

Step 4: According to the aesthetic style to be enhanced, perform the corresponding degenerate transformation on the real image I and the inversely mapped image I _inv and calculate the loss, and then optimize the hidden vector zi _,(i∈(1,n)) according to gradient descent, Take the optimized zi _{, (i∈(1,n))} as input again, obtain a new inversely mapped image I _inv through the mGANprior method in step 3, and calculate the loss until the loss has no downward trend for ten consecutive iterations. Stop training and get the final latent vector zi _,(i∈(1,n)) and the image I _enh after image aesthetic style enhancement. The loss function adopts the combination of mean square error loss (MSE) and perceptual feature reconstruction loss.

Further, the specific method of step 3 is as follows:

The mGANprior method is used to divide the generation network of the generation model selected in step 1 into two from the specified layer. This layer and the previous network layer are the pre-network G1, and all the networks after this layer are the post-network G2. The specified layer The level is selected according to the needs, and the effect of inverse mapping is positively related to the depth of the level. Randomly generate a number of n latent vectors z _{i, (i∈(1,n))} as the input of the pre-network G1, obtain n feature maps through the pre-network G1, and combine the obtained feature maps based on the adaptive channel importance combination Then input the post network G2 to get the generated image I _inv .

Further, the specific method of step 4 is as follows:

(1) Apply image degradation transformation to the real image I and the inversely mapped image I _inv to obtain X and X _inv .

If the beauty of shallow depth of field is enhanced, do the following shallow depth of field degradation transformation. Downsample the pixels in the background part of the real image I and the main part of the inverse-mapped image I _inv according to m. The degenerate transformation formula is as follows:

X=I*m+down(I*1-m))#(1)

X _inv =down(I _inv *m)+I _inv *(1-m)#(2)

If the beauty of motion blur is enhanced, the following motion blur degradation transformation is performed. Downsample the pixels of the main part of the real image I according to m. The degenerate transformation formula is as follows:

X=down(I*m)+I*(1-m)#(3)

X _inv =I _inv #(4)

(2) Calculate the MSE and perceptual feature loss of X and X _inv :

为X与X_inv的MSE损失，其中φ(·)为感知特征提取器，φ(X)和φ(X_inv)分别为X与X_inv的感知特征，||φ(X),φ(X_inv)||₁为φ(X)和φ(X_inv)的L1距离。in

is the MSE loss of X and X _inv , where φ( ) is the perceptual feature extractor, φ(X) and φ(X _inv ) are the perceptual features of X and X _inv , respectively, ||φ(X),φ(X _inv )|| ₁ is the L1 distance of φ(X) and φ(X _inv ).

(3) Use gradient descent to optimize zi _,(i∈(1,n)) .

(4) Iterative training. Using the mGANprior method of step 3, the optimized zi _{, (i∈(1,n))} is used as the input of the pre-network G1 again, and n feature maps are obtained; according to the principle of adaptive channel importance, the n feature maps are Perform combined input G2 to obtain a new inversely mapped image I _inv ;

(5) Repeat the above steps (1)-(4) to do degenerate transformation and calculate the loss, optimize zi _,(i∈(1,n)) according to gradient descent and iteratively train. The training is stopped until the loss no longer has a downward trend for ten consecutive iterations, and the I _enh with enhanced fuzzy aesthetics is obtained. Using the shallow depth-of-field degradation transform results in a shallow depth-of-field aesthetic-enhanced image, and using the motion-blur degradation transform results in a motion-blurred aesthetic-enhanced image.

The beneficial effects of the present invention are as follows:

1. Using mGANprior, an inverse mapping method of GAN, to enhance the beauty of real images. It not only preserves the original information of the image to the greatest extent, but also modifies the aesthetic style of the image in a controllable manner.

2. The blur effect in the aesthetic style, such as shallow depth of field and motion blur, is the most difficult effect of the two GANs, because the GAN model will try to eliminate the blur during the training process. This patent proposes a loss of degenerate transformation based on aesthetic factors. function to generate an aesthetically pleasing blur effect on a real image.

Description of drawings

Fig. 1 is a flow chart of the implementation of the method of the present invention.

Detailed ways

The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

As shown in Figure 1, the present invention proposes a real image aesthetic enhancement method based on mGANprior, input a real image and select a suitable PGGAN generation model, and perform semantic segmentation on the image to extract the main part to be enhanced. The extracted semantics generate a binary matrix m, where the value of the main part is 1 and the value of the background part is 0. Use mGANprior to obtain the inversely mapped image, degenerate the real image and the inversely mapped image and calculate the loss, and perform parameter optimization and iterative training according to gradient descent, and finally obtain an image with enhanced aesthetics. The detailed steps are as follows.

Step 1: Select a real image I to be enhanced with aesthetics, and manually determine whether the image belongs to a certain category in the PGGAN pre-training generation model. For example, for images containing bridges, select the PGGAN corresponding bridge generation model. If it does not belong to any of the PGGAN pre-trained generative models, a PGGAN generative model can be selected arbitrarily. And decide whether you want a shallow depth of field aesthetic enhancement or a motion blur aesthetic enhancement to this image. .

Step 2: Semantically segment the image using a cascaded segmentation module to extract subject pixels. And the binary matrix m is obtained from this. If the beauty of shallow depth of field is enhanced, the part that needs to be clearly imaged is set to 1 in m, and the rest of the values are set to 0; if the beauty of motion blur is enhanced, the blurred part needs to be added in the value of m. Set to 1, other values are set to 0.

The depth of field refers to the range of clear imaging before and after the focus point. The picture elements within this range can be clearly imaged, while the picture elements outside the range will gradually become blurred. Shallow depth of field means that only part of the image is in focus. Motion blur is the apparent blurring of a static scene or a series of pictures like a movie or animation of fast-moving objects.

Step 3: Obtain the inverse mapping image I _inv through the mGANprior model.

(1) The PGGAN generation network takes the 8th layer as the dividing line, the first layer to the eighth layer are used as the front network G1, and the subsequent network is used as the rear network G2;

(2) Generate 30 hidden vectors z _{i,(i∈(1,30))} ;

(3) z _{i, (i∈(1,30))} input G1 to get 30 feature maps;

(4) 30 feature maps are combined according to the principle of adaptive channel importance and input to G2 to obtain an inversely mapped image I _inv .

Step 4: Apply different degradation transformations to the real image I and the inversely mapped image I _inv according to the aesthetic feeling to be enhanced to obtain X and X _inv , calculate the loss of X and X _inv as the training loss, and optimize zi _{, (i} by gradient descent _∈(1,30)) .

(1) Apply image degradation transformation to the real image I and the inversely mapped image I _inv to obtain X and X _inv .

If the beauty of shallow depth of field is enhanced, do the following shallow depth of field degradation transformation. Downsample the pixels in the background part of the real image I and the main part of the inverse-mapped image I _inv according to m. The degenerate transformation formula is as follows:

X=I*m+down(I*(1-m))#(1)

X _inv =down(I _inv *m)+I _inv *(1-m)#(2)

If the beauty of motion blur is enhanced, the following motion blur degradation transformation is performed. Downsample the pixels of the main part of the real image I according to m. The degenerate transformation formula is as follows:

X=down(I*m)+I*(1-m)#(3)

X _inv =I _inv #(4)

(2) Calculate the MSE and perceptual feature loss of X and X _inv

为X与X_inv的MSE损失，其中φ(·)为感知特征提取器，φ(X)和φ(X_inv)分别为X与X_inv的感知特征，||φ(X),φ(X_inv)||₁为φ(X)和φ(X_inv)的L1距离。in

is the MSE loss of X and X _inv , where φ( ) is the perceptual feature extractor, φ(X) and φ(X _inv ) are the perceptual features of X and X _inv , respectively, ||φ(X),φ(X _inv )|| ₁ is the L1 distance of φ(X) and φ(X _inv ).

(3) Use gradient descent to optimize zi _{,(i∈(1,30))} .

(4) Iterative training. Using the mGANprior method in step 3, the optimized zi _{, (i∈(1,30))} is used as the input of the pre-network G1 again, and 30 feature maps are obtained; according to the principle of adaptive channel importance, the 30 feature maps are Perform combined input G2 to obtain a new inversely mapped image I _inv ,

(5) Repeat the above steps (1)-(4) to do degenerate transformation and calculate the loss, optimize zi _{,(i∈(1,30))} according to gradient descent and iteratively train. The training is stopped until the loss no longer has a downward trend for ten consecutive iterations, and the I _enh with enhanced fuzzy aesthetics is obtained. Using the shallow depth-of-field degradation transform results in a shallow depth-of-field aesthetic-enhanced image, and using the motion-blur degradation transform results in a motion-blurred aesthetic-enhanced image.

Claims (3)

1. A method for enhancing the aesthetic feeling of a real image based on mGAN prior is characterized by comprising the following steps:

step 1: selecting a real image I to be enhanced in aesthetic feeling, selecting a PGGAN pre-training generation model of a corresponding type, and determining that shallow depth of field aesthetic feeling enhancement or motion blur aesthetic feeling enhancement needs to be performed on the image;

step 2: performing semantic segmentation on the real image I by using a cascade segmentation module method, and extracting a main pixel; obtaining different binary matrixes m according to the aesthetic feeling effect enhanced by the requirement;

step (ii) of3: obtaining an inverse mapped image I by using an mGANprepror method_inv；

And 4, step 4: according to the aesthetic style to be enhanced, the real image I and the inverse mapping image I are subjected to_invMaking a corresponding degenerate transformation and calculating the loss, and then optimizing the hidden vector z according to the gradient descent_{i，(i∈(1，n))}Will optimize z_{i，(i∈(1，n))}Again as input, a new inverse mapped image I is obtained by the mGANPrior method of step 3_invAnd calculating loss, stopping training until ten continuous iterations of the loss have no descending trend, and obtaining a final hidden vector z_{i，(i∈(1，n))}Image I with enhanced aesthetic style_enh(ii) a The loss function adopts a mode of combining mean square error loss (MSE) and perceptual feature reconstruction loss.

2. The method for enhancing the real image aesthetic feeling based on the mGANprior as claimed in claim 1, wherein the specific method in step 3 is as follows:

dividing the generation network of the generation model selected in the step 1 into two parts from a designated layer by adopting an mGANNprior method, wherein the layer and the previous network layer are front networks G1, all the networks behind the layer are rear networks G2, the designated layer is selected by self according to requirements, and the inverse mapping effect is positively correlated with the layer depth; randomly generating a number n of hidden vectors z_{i，(i∈(1，n))}The pre-network G1 is used as input of the pre-network G1 to obtain n feature maps, the obtained feature maps are combined based on the adaptive channel importance and input into the post-network G2 to obtain a generated image I_inv。

3. The method for enhancing the real image aesthetic feeling based on the mGANprior as claimed in claim 1, wherein the step 4 is as follows:

(1) for real image I and inverse mapped image I_invApplying an image degradation transformation to obtain X and X_inv；

If the aesthetic feeling of the shallow depth of field is enhanced, performing the following shallow depth of field degradation transformation; according to m pairs of real image I background part and inverse mapping image I_invDown-sampling the pixels of the middle main body part; the degenerate transformation equation is as follows:

X＝I*m+down(I*(1-m))#(1)

X_inv＝down(I_inv*m)+I_inv*(1-m)#(2)

if the aesthetic feeling of the motion blur is enhanced, performing the following motion blur degradation transformation; downsampling the pixels of the main part of the real image I according to the m; the degenerate transformation equation is as follows:

X＝down(I*m)+I*(1-m)#(3)

X_inv＝I_inv#(4)

(2) calculating X and X_invMSE and perceptual feature loss of:

wherein

Is X and X_invWhere φ (-) is a perceptual feature extractor, φ (X) and φ (X)_inv) Are X and X respectively_invThe perceptual features of (1), i (X), phi (X)_inv)||₁Is phi (X) and phi (X)_inv) L1 distance;

(3) optimizing z with gradient descent_{i，(i∈(1，n))}；

(4) Performing iterative training; optimizing z by using the method of step 3 of mGANPrior_{i，(i∈(1，n))}Taking the n characteristic graphs as the input of the preposed network G1 again; combining the n characteristic graphs according to the principle of the importance of the self-adaptive channel and inputting the n characteristic graphs into G2 to obtain a new image I of inverse mapping_inv；

(5) Repeating the steps (1) to (4) to perform degradation transformation and calculate loss, and performing gradient descent on z_{i，(i∈(1，n))}Optimizing and performing iterative training; stopping training until ten consecutive iterations are lost and no longer trend downward, obtaining fuzzy aesthetic enhanced I_enh(ii) a Obtaining a shallow depth of field aesthetically enhanced image using a shallow depth of field degenerative transform, using motion blur degradationThe transformation results in a motion blurred, aesthetically enhanced image.

Images