CN115272632B - Virtual fitting method based on gesture migration - Google Patents

Abstract

The present invention relates to a virtual fitting method based on posture migration, comprising: obtaining an original analysis image and an image of a person trying on clothes, first extracting clothing pixel information in the image of the person trying on, and then performing texture restoration to obtain a fine clothing image; converting the original analysis image Input the target posture and the target pose into the analysis guide network to obtain the analysis guide map; according to the analysis guide map, make a preliminary limit on the distortion range of the clothing; obtain the target pose and preprocess to obtain the analysis map with the lower body removed, and obtain the distortion through the clothing warp network The clothing image of the image; according to the analysis guide map, the distorted clothing image, the target pose and the image of the tryer, the try-on result of the target pose is generated. The present invention simultaneously inputs the analysis guide map, the distorted target clothing image, the target posture and the image of the try-on person to the neural network model to obtain the fitting effect diagram of the target posture, which improves the fitting effect and solves the problem caused by the transformation of the try-on posture. Issue with skin and cloth pixels getting mixed up.

Description

Virtual fitting method based on pose transfer

technical field

The invention belongs to the field of clothing image processing, in particular to a virtual fitting method based on posture transfer.

Background technique

In recent years, with the transformation of shopping methods from offline to online, online clothing shopping has been favored by consumers. However, there is a problem that consumers cannot try on clothes, and consumers cannot experience the effect of clothing on themselves. The emergence of virtual fittings can enable the seller to display the advantages of clothing more objectively, so that both parties to the transaction can understand information more intuitively, facilitate transactions, reduce unnecessary work, improve work efficiency, and meet user needs.

At present, the virtual fitting method of the existing technology that combines virtual fitting and posture migration to realize multi-pose virtual fitting is mainly divided into two methods based on 2D images and 3D reconstruction. For multi-pose fitting methods directly based on 2D images There are relatively few technologies, and there are skin and cloth pixel confusion and loss of details in the fitting results; the effect based on 3D reconstruction is better, but the requirements for equipment computing power, performance and the quality of the generated model are relatively high, which is not conducive to Promotion and popularization of technology.

The Chinese patent with the publication number CN 108734787A discloses "a virtual fitting method for picture synthesis based on multi-pose and part decomposition", which is synthesized by using multi-pose and part decomposition instead of simply taking the whole clothes picture Simple synthesis can more realistically achieve the effect of virtual fitting, but this technology does not consider the confusion of skin and cloth pixels and loss of details caused by pose changes, which greatly affects the fitting effect.

Contents of the invention

The purpose of the present invention is to address the above-mentioned problems, provide a kind of virtual fitting method based on posture transfer, utilize the analytical guide map to limit the distortion range of target clothing image, avoid excessive distortion when target clothing image changes with target posture; According to target posture And remove the analysis map of the lower body, use the clothing warping network to obtain the target clothing image distorted with the target posture, input the try-on image, the target clothing image distorted with the target posture, the target posture and the analysis guide map at the same time to generate Network, using the try-on image generation network to obtain the try-on result of the target pose, improve the try-on effect, avoid the problem of skin and cloth pixel confusion caused by the change of try-on pose, and maintain more clothing texture details.

The technical solution of the present invention is a virtual fitting method based on posture migration, comprising the following steps:

Step 1. Obtain the original analysis image and the image of the wearer, first extract the clothing pixel information in the image of the tryer to obtain a simple clothing image, and then perform texture restoration to obtain a fine clothing image;

Step 2, input the original analytical map, target clothing and target pose into the analytical guidance network to obtain the analytical guidance map;

Step 3: Preliminarily limit the distortion range of the target clothing according to the analysis guide map;

Step 4: Obtain the target pose and preprocess it to obtain the analytical image with the lower body removed, and obtain the distorted target clothing image through the clothing warping network;

Step 5: According to the analysis guide map, the distorted target clothing image, the target pose and the image of the person trying on the clothes, the try-on result of the target pose is generated through the image generation network.

Further, step 1 performs pixel-level repair on the clothing image, and the specific repair process includes:

First learn the edge information features of the clothing image through the convolutional neural layer, and focus on the area where the pixel value changes drastically; then use the interpolation method to repair the pixel in the area where the pixel value changes rapidly, to ensure that the edge of the clothing is smooth and transitions naturally with the background.

Preferably, step 1 includes the following sub-steps:

First, according to the clothing semantic information in the original analysis image, the pixel information of the corresponding area in the try-on image is extracted to obtain a preliminary clothing image. The clothing image has blurred image edges or gaps in the image edges;

Then, the interpolation method is used to fill or fill the blurred and gap areas in the clothing image to obtain a more detailed clothing image.

Further, the specific process of step 2 is as follows:

First, input the original analysis image and target pose into the analysis-guided network, use the multi-layer convolutional network of the analysis-guided network to extract image features, and add a residual module and wavelet sampling layer in the analysis-guided network to extract more advanced The semantic structure enables the analysis-guided network to deeply learn the details of the relationship between various parts of the human body. Among them, the wavelet sampling layer converts the feature map into the frequency domain through wavelet transform for down-sampling, which can better retain texture information;

Then, the extracted image features are input into the multi-layer deconvolution network of the analytical guidance network, the image is up-sampled, and a normalization layer is added between deconvolutions to enhance the relationship between global features and local features. Feature fusion among them, and introduce a normalized constraint loss function to control the retention of more semantic details in the upsampling process;

Finally, compare the spatial position of the generated analytic guide map with the target state to ensure that each semantic part fits in position with the corresponding posture key points, better handle the overlapping relationship between the arm and the clothing, and correct the semantic position Perform fine-tuning to obtain a more regular analytical guide map.

Preferably, the normalized constraint loss function is as follows:

In the formula, Represents the normalized constraint loss function, G represents the global feature of the image, G' represents the global feature of the analyzed image, L represents the local feature of the image, L' represents the local feature of the parsed image, /> Represents the image global feature matching loss function before and after parsing, /> Represents the image local feature matching loss function before and after parsing, Both are learning coefficients, which are used to adjust the importance of global features and local features.

The parsing guide map contains semantic segmentation information, including: face, hair, neck, upper garment area, left arm, right arm, left hand, right hand, left shoulder, right shoulder, lower garment area.

Preferably, the target pose contains 18 key points, specifically including: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, Left ankle, right eye, left eye, right ear, left ear.

Further, the specific process of step 4 is as follows:

First, remove the semantic information of the lower body according to the difference in pixel values of each semantic information in the analysis guide map, and obtain the analysis map with the lower body removed;

Then, by removing the analysis map and the target pose of the lower body, the overall contour of the clothing image is limited, and the warping network is prevented from forcibly deforming the clothing image, avoiding excessive distortion of the clothing;

Finally, the clothing image is deformed through the warping network and the plane deformation loss function is introduced to obtain the warped clothing image.

Preferably, the plane deformation loss function is as follows:

In the formula, C _x (x), C _y (x) represent the x and y coordinates of the sampling parameters respectively, and |C _x (x+i,y)-C _x (x,y)| represents the distance between two nodes Euclidean distance, i, j are deformation variables, γ, δ are deformation coefficients.

Further, in step 5, the try-on image generation network is an end-to-end network, which includes a generator and a discriminator. The input of the generator is the analysis guide map, the distorted clothing image and the try-on image. Under the constraints, a rough try-on result map is generated according to the pixel information of the distorted clothing image and the try-on image, and then the discriminator and the feature point loss function are introduced to determine whether the rough try-on result map conforms to the target pose and extract more arms Regional features, continuously enhance the details of the rough try-on result image, and improve image clarity.

Preferably, the feature point matching loss function is as follows:

In the formula, Represents the feature point matching loss function, W represents the coordinate point of the human body posture in the rough try-on result image, M represents the coordinate point of the target posture, W _i (x) represents the abscissa of the coordinate point i in the rough try-on result graph, M _i ( x) represents the abscissa of the coordinate point i in the target posture graph, n represents the total number of feature points, |W _i (x)-M _i (x)| represents the Euclidean distance of the key points of the same part on the x-axis, α, β Both are adjustment coefficients, and α+β=1.

Compared with the prior art, the beneficial effects of the present invention include:

(1) In the present invention, the analysis guide map containing semantic segmentation information, the target clothing image distorted with the target posture, the target posture and the image of the try-on person are input into the try-on image generation network at the same time, and the try-on image generation network is used to obtain the try-on The fitting effect picture of the target pose of the user, which greatly improves the fitting effect, solves the problem of confusion between the skin and the cloth pixels caused by the change of the try-on pose, and maintains more clothing texture details in the fitting effect map, which improves the virtual The experience of trying on.

(2) The present invention limits the distortion range of the target clothing image by using the analysis guide map containing semantic segmentation information, avoiding excessive distortion of the target clothing image when changing with the target posture, and making the virtual try-on effect more realistic.

(3) The present invention obtains the clothing image from the image of the wearer, and refines the texture of the blur and gap regions in the clothing image to obtain a finer clothing image, which solves the problem of lack of clothing images in the training data set, and helps The robustness of the fitting method is enhanced by strengthening the training of the try-on image generation network, the analysis-guided network, and the clothing warping network.

(4) The present invention introduces a normalization layer and a normalization constraint loss function during the analysis guidance process of obtaining the analysis guidance map, and while enhancing the fusion of global features and local features, more features are retained in the control upsampling process Semantic details.

(5) The present invention introduces a feature point matching loss function in the try-on image generation network to determine whether the preliminary try-on result map conforms to the target posture, effectively avoids the problem of cross-occlusion of arms and clothing, and further improves the effect of virtual fitting.

Description of drawings

The present invention will be further described below in conjunction with drawings and embodiments.

FIG. 1 is a schematic flowchart of a virtual fitting method according to an embodiment of the present invention.

Fig. 2 is a structural diagram of the analytical guidance network of the virtual fitting method according to the embodiment of the present invention.

Fig. 3 is a structural diagram of the warping network of the virtual fitting method of the embodiment of the present invention.

Fig. 4 is a network structure diagram of a virtual fitting image generation network of a virtual fitting method according to an embodiment of the present invention.

Fig. 5 is a schematic diagram of a virtual fitting system according to an embodiment of the present invention.

Detailed ways

Embodiment one

As shown in Figure 1, the virtual fitting method based on gesture migration includes the following steps:

(1) Obtain the original analysis image and the image of the wearer, first extract the clothing pixel information in the image of the tryer to obtain a simple clothing image, and then perform texture restoration to obtain a fine clothing image;

Among them, the acquisition process of the clothing image is as follows: First, according to the clothing semantic information in the original analysis image, the pixel information of the corresponding area in the try-on image is extracted to obtain a rough clothing image with blurred edges and gaps. Then, the texture of the clothing image is repaired, and the blurred and gap areas in the rough clothing image are filled or filled by interpolation method to obtain a finer clothing image.

The original parsing image contains semantic information of each part of the wearer, including: face, hair, neck, upper garment area, left arm, right arm, left hand, right hand, left shoulder, right shoulder, and lower garment area.

Among them, the texture repair first learns the edge information characteristics of the clothing image through the convolutional neural network, focuses on the area where the pixel value changes drastically, and then uses the interpolation method to perform pixel repair on the area where the pixel value changes sharply to ensure that the edge of the clothing is smooth and natural to the background. transition.

(2) Obtain the original analytical map and the target pose, input it to the analytical guidance network, and obtain the analytical guidance map;

Among them, the analytical guide map shows the semantic segmentation information after the pose transformation of the try-on user, including information on the face, hair, neck, top, arm, and bottom.

The target pose consists of 18 key points, including nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right Eye, left eye, right ear, left ear.

As shown in Figure 2, the parsing-guiding network consists of a multi-layer convolutional network and a multi-layer deconvolution network. The input of the parsing-guiding network is the original parsing map, target clothing and target pose, and the output is the parsing-guiding map.

Among them, the analysis guidance process is as follows: First, input the original analysis image and target pose, extract image features through a multi-layer convolutional network, and add a residual module and wavelet sampling layer to the analysis guidance network to extract higher-level semantics The structure enables the analysis-guided network to deeply learn the details of the relationship between various parts of the human body. Among them, the wavelet sampling layer converts the feature map to the frequency domain through wavelet transform for down-sampling, which can better retain texture information; then, the extracted The image features of the image are input into the multi-layer deconvolution network, the image is upsampled, and a normalization layer is added between deconvolutions to enhance the feature fusion between global features and local features, and introduce normalization The constraint loss function is optimized to control the preservation of more semantic details during the upsampling process. Finally, compare the spatial position of the generated analytic guide map with the target state to ensure that each semantic part fits in position with the corresponding posture key points, better handle the overlapping relationship between the arm and the clothing, and correct the semantic position Perform fine-tuning to obtain a more regular analytical guide map.

Among them, in the normalization layer, the features obtained by the previous layer of deconvolution are regarded as local features, and the features obtained by the latter layer of deconvolution are regarded as global features, and the current local features are controlled by introducing a normalization constraint loss function and the impact of global features on subsequent fusion results.

Among them, the normalized constraint loss function is expressed as:

In the formula, Represents the normalized constraint loss function, G represents the global feature of the image, G' represents the global feature of the analyzed image, L represents the local feature of the image, L' represents the local feature of the parsed image, /> Represents the image global feature matching loss function before and after parsing, /> Represents the image local feature matching loss function before and after parsing, Both are learning coefficients, which are used to adjust the importance of global features and local features.

(3) According to the analysis guide map, make a preliminary limit on the distortion range of the clothing;

(4) Obtain the target pose and preprocess it to obtain the analytical image with the lower body removed, and obtain the distorted clothing image through the clothing warping network, as shown in Figure 3;

Among them, the specific process of obtaining the distorted clothing image is as follows: First, according to the difference in the semantic information pixel value of the analysis guide image, the semantic information of the lower body is removed to obtain the analysis image with the lower body removed; then, by removing the analysis image of the lower body and The target pose limits the overall outline of the clothing image distortion, avoiding the warping network to forcibly deform the clothing image, and avoiding excessive distortion of the clothing; finally, the warping network and the introduction of the plane deformation loss function are used to deform the clothing image to obtain the distorted clothing image.

Among them, the plane deformation loss function is expressed as:

In the formula, C _x (x), C _y (x) represent the x and y coordinates of the sampling parameters respectively, and |C _x (x+i,y)-C _x (x,y)| represents the distance between two nodes Euclidean distance, i, j are deformation variables, γ, δ are deformation coefficients.

(5) According to the analysis guide map, the distorted clothing image, the target pose and the image of the try-on person, the try-on result of the target pose is generated through the image generation network;

As shown in Figure 4, the image generation network is an end-to-end network consisting of a generator and a discriminator. The generator is composed of an encoder and a decoder. The image of the wearer, under the limitation of the analysis guide map, generates a rough try-on result map according to the pixel information of the distorted clothing image and the try-on image, and obtains the human body pose map of the rough try-on result map, and then passes through the discriminator and introduces The feature point matching loss function determines whether the rough try-on result map conforms to the target pose, and encourages the extraction of more arm region features, continuously enhances the details of the rough try-on result map, and improves image clarity.

Among them, the feature point matching loss function is as follows:

In the formula, Represents the feature point matching loss function, W represents the coordinate point of the human body posture in the rough try-on result image, M represents the coordinate point of the target posture, W _i (x) represents the abscissa of the coordinate point i in the rough try-on result graph, M _i ( x) represents the abscissa of the coordinate point i in the target posture graph, n represents the total number of feature points, |W _i (x)-M _i (x)| represents the Euclidean distance of the key points of the same part on the x-axis, α, β Both are adjustment coefficients, and α+β=1.

Embodiment two

As shown in Figure 5, the virtual fitting system for posture transfer includes an analysis guidance module, a clothing matching module, and an image fusion module.

The analysis guidance module is used to perform pixel extraction and texture restoration according to the original analysis image, the image of the wearer and the target pose, and then generate the analysis guidance image through the analysis guidance network;

The clothing matching module is used to obtain the distorted clothing image through the clothing warping network according to the analysis guide map, the target pose and the analysis map with the lower body removed;

The image fusion module generates the try-on result of the target pose through the try-on image generation network according to the analysis guide map, the distorted clothing image, the target pose and the try-on image.

As shown in Figure 2, the input of the semantic parsing network is the original parsing map and the target pose map, and the output is the parsing guide map, that is, the parsing map after pose migration. The original parsing image and the target pose image are respectively processed by sequentially connected five residual blocks, and each residual block uses 3×3 convolution to extract features, and the residual blocks are connected by a wavelet layer. Downsample the feature map; the residual block at the end is connected to the normalization layer to enhance the feature fusion between global features and local features, and introduce a normalized constraint loss function to control more semantics during the upsampling process Details; After normalization, it is processed by 5 deconvolution layers connected in sequence, adjacent deconvolution layers are connected by inverse wavelet layer, the inverse wavelet layer is used for upsampling, and the final deconvolution layer outputs analytical guidance picture.

As shown in Figure 3, the input of the clothing warping network is the parsing guide map and the clothing image, and the output is the warped clothing image. First, the analysis guide map and the clothing image are encoded by the encoder, and the image features of the two are extracted respectively; then, the deformation coefficient θ is calculated through the image features of the two, and the distortion of the clothing image is limited by analyzing the guide map and the target pose. The overall outline avoids the warping network from forcibly deforming the clothing image and avoiding excessive distortion of the clothing; finally, the clothing image is deformed by the warping operation and the introduction of the plane deformation loss function to obtain the distorted clothing image.

As shown in Figure 4, the input of the try-on image generation network is the parsing guide map, the distorted clothing image and the try-on image, and the output is the try-on image. The try-on image generation network is an end-to-end network. It includes a generator and a discriminator. The generator is composed of an encoder and a decoder. Under the limitation of the guide map, the rough try-on result map is generated according to the pixel information of the distorted clothing image and the try-on image, and then the discriminator and the feature point loss function are introduced to determine whether the rough try-on result map conforms to the target pose, and extract More arm region features, enhanced details of the rough try-on result image, and improved image clarity.

The virtual fitting system for posture migration adopts the same virtual fitting method as that of the first embodiment.

The implementation results show that the invention not only makes the accuracy of semantic segmentation higher, but also increases the robustness of clothing deformation, makes the try-on result image retain more details, greatly improves the virtual try-on effect of high-resolution 2D images, and improves the Try-on effect and user experience.

Claims (6)

1. The virtual fitting method based on gesture migration is characterized by comprising the following steps of:

step 1, acquiring an original analysis image and a try-on image, firstly extracting clothing pixel information in the try-on image to obtain a crude clothing image, and then performing texture repair to obtain a fine clothing image;

step 2, inputting the original analysis chart, the target clothing and the target gesture into an analysis guiding network to obtain an analysis guiding chart;

step 3, primarily limiting the twisting range of the target clothing according to the analysis guide diagram;

step 4, acquiring a target gesture, preprocessing to obtain an analysis chart from which the lower body is removed, and obtaining a distorted clothing image through a clothing warping network;

step 5, generating a try-on result of the target gesture through a try-on image generating network according to the analysis guide map, the distorted target clothing image, the target gesture and the try-on image;

the specific process of the step 2 is as follows:

firstly, inputting an original analysis chart and a target gesture into an analysis guiding network, extracting image features by utilizing a multi-layer convolution network of the analysis guiding network, adding a residual error module and a wavelet sampling layer into the analysis guiding network for extracting higher-level semantic structures, so that the analysis guiding network deeply learns the relation details among all parts of a human body, wherein the wavelet sampling layer converts the feature chart into a frequency domain for downsampling through wavelet transformation, and texture information can be better reserved;

then, inputting the extracted image features into a multi-layer deconvolution network of an analysis guide network, up-sampling the image, adding a normalization layer between convolution and deconvolution to enhance feature fusion between global features and local features, introducing a normalization constraint loss function, and controlling the up-sampling process to retain more semantic details;

finally, comparing the space position of the generated analysis guide graph with that of the target state, ensuring that each semantic part is attached to the corresponding gesture key point in position, better processing the overlapping relationship between the arm and the garment, and fine-tuning the semantic position to obtain a more regular analysis guide graph;

the semantic analysis network comprises 5 residual blocks which are sequentially connected, wherein the residual blocks are connected through a wavelet layer, and the wavelet layer downsamples the feature map in a frequency domain space; the residual block at the tail end is connected with a normalization layer and is used for enhancing feature fusion between global features and local features, introducing a normalization constraint loss function and controlling more semantic details to be reserved in the up-sampling process; after normalization, processing by 5 deconvolution layers which are sequentially connected, connecting adjacent deconvolution layers by an inverse wavelet layer, wherein the inverse wavelet layer is used for up-sampling, and outputting an analysis guide graph by the deconvolution layer at the tail end;

in step 5, the test image generation network comprises a generator and a discriminator, wherein the input of the generator is an analysis guide image, a distorted clothing image and a test person image, a rough test result image is generated according to pixel information of the distorted clothing image and the test person image under the limitation of the analysis guide image, and then the rough test result image is judged whether to conform to the target gesture or not through the discriminator and the feature point matching loss function is introduced, and more arm region features are extracted, so that the details of the rough test result image are enhanced, and the image definition is improved;

the feature point matching loss function is as follows:

in the method, in the process of the invention,representing a characteristic point matching loss function, W representing a human body posture coordinate point in a rough test result graph, M representing a target posture coordinate point, W _i (x) Representing the abscissa of a human body posture coordinate point i in a rough test result diagram, M _i (x) The abscissa of coordinate point i in the target gesture graph is represented, n represents the total number of characteristic points, and W is represented _i (x)-M _i (x) And the I represents the Euclidean distance of the key point of the same part on the x axis, alpha and beta are adjustment coefficients, and alpha+beta=1.

2. The virtual fitting method according to claim 1, wherein step 1 performs a pixel level repair of the garment image, and the specific repair process includes: firstly, learning edge information characteristics of a clothing image through a convolutional neural layer, and focusing on a region with a pixel value which is changed severely; and then, carrying out pixel repair on the region with the pixel value which is changed severely by using an interpolation method, so as to ensure that the clothing edge is smooth and is in natural transition with the background.

3. A virtual fitting method according to claim 2, characterized in that step 1 comprises the sub-steps of:

firstly, extracting pixel information of a corresponding region in an image of a test wearer according to clothing semantic information in an original analytic graph to obtain a rough clothing image, wherein the clothing image has the condition of blurred image edges or gaps on the image edges;

and then, carrying out texture restoration on the clothing image, and carrying out pixel filling or filling on the fuzzy and notch areas in the rough clothing image by using an interpolation method to obtain a finer clothing image.

4. A virtual fitting method according to claim 3, wherein the normalized constraint loss function is as follows:

in the method, in the process of the invention,representing a normalized constraint loss function, G represents a global feature of the image, G ^′ Representing global features of the resolved image, L representing local features of the image, L ^′ Representing local features of the parsed image, +.>Representing the global feature matching loss function of the image before and after parsing, < + >>Represents the local feature matching loss function of the image before and after analysis,are learning coefficients.

5. A virtual fitting method according to claim 3, wherein the specific procedure of step 4 is as follows:

firstly, removing semantic information of the lower body according to different pixel values of semantic information in the analysis guide graph to obtain an analysis graph from which the lower body is removed;

then, by removing the analytic graph and the target gesture of the lower body, the whole outline of the distortion of the clothing image is limited, the forced deformation of the clothing image by the warping network is avoided, and the excessive distortion of the clothing is avoided;

and finally, deforming the clothing image through a warping network and introducing a plane deformation loss function to obtain a distorted clothing image.

6. The virtual fitting method according to claim 5, wherein the plane deformation loss function is as follows:

wherein C is _x (x),C _y (x) Respectively representing x and y coordinates, |C of the sampling parameter _x (x+i,y)-C _x (x, y) I represents Euclidean distance between two nodes, i, j are deformation amounts, and gamma, delta are deformation coefficients.