CN110288677A - A Pedestrian Image Generation Method and Device Based on Deformable Structure - Google Patents

Abstract

The invention relates to the field of image generation, in particular to a pedestrian image generation method and device based on a deformable structure. It specifically includes the following steps: step 1, segment the pedestrian picture and the target pose picture according to the part structure, and extract the mask operation; step 2, then perform the part generation operation, and obtain the part generated picture; step 3, perform structure on the part generated picture Combine operations to obtain structured merged pictures; step 4, perform overall generation operations to obtain generated pictures. The invention reduces the cost of training and improves the performance of the algorithm on the basis of considering the deformable structure of the human body.

Description

A Pedestrian Image Generation Method and Device Based on Deformable Structure

technical field

The invention relates to the field of image generation, in particular to a pedestrian image generation method and device based on a deformable structure.

Background technique

Converting from a pedestrian image to another pedestrian image according to a given pose is a pedestrian image generation problem. The problem of pedestrian image generation is a field of image generation. Compared with ordinary image generation, pedestrian image generation will be more complex and challenging because it needs to consider more complex scenes and various deformable poses.

The problem of pedestrian image generation can be solved according to the traditional image generation idea, such as using conditional confrontation generation network, using the source image of the whole body of the human body as a condition to guide the network to generate a new pose image with the appearance of the source image; it is also possible to use a cyclic confrontation generation network to replace The background and lighting of pedestrian pictures, on the basis of retaining human characteristics, generate new poses and pedestrian pictures in the environment. The biggest problem with this method is that it is difficult to train. The human body is too complex as a deformable object, and the complex image conversion relationship requires a very large number of training samples.

It is a better solution to introduce human body information into the generation process, such as using posture information as part of the input information to provide guidance on prior conditions. The key to the deformable complexity of the human body is the diversity of poses. The prior guidance of pose information can effectively alleviate the generation complexity, so that more realistic pedestrian pictures can be generated. The same problem still exists, the posture transformation of the whole body is still complicated, and a large number of training samples are still required to generate more realistic pictures.

Contents of the invention

Embodiments of the present invention provide a method and device for generating pedestrian images based on deformable structures, which reduce the cost of training and improve the performance of the algorithm on the basis of considering the deformable structure of the human body.

According to the first aspect of the embodiments of the present invention, the present invention provides a pedestrian image generation method based on a deformable structure, which specifically includes the following steps:

Step 1. For the input pedestrian pictures and target pose pictures, the pedestrian pictures and target pose pictures are segmented according to the part structure, and the obtained part pedestrian pictures and part target pose pictures are divided into pedestrian pictures, target pose pictures, part pedestrian pictures and The target pose pictures of the parts are all subjected to the mask operation to obtain the pedestrian mask picture, the target pose picture mask picture, the part pedestrian mask picture and the part target pose picture mask picture;

Step 2, preprocessing the part pedestrian picture, and then performing the part generation operation on the preprocessed part pedestrian picture, part target pose picture and part target pose mask picture, to obtain the part generation picture;

Step 3, performing a structured merging operation on the part generated picture obtained by the part generating operation in step 2, to obtain a structured merging picture;

Step 4: Perform preprocessing on the original pedestrian picture, use the preprocessed pedestrian picture, the combined picture in step 3, and the target pose picture as input, and then perform the overall generation operation to obtain the generated picture.

In the first step, the segmentation operation specifically includes the following steps:

1.1 For pedestrian pictures and target pose pictures, use the joint point detection algorithm to find the joint points of the input picture;

1.2 Judging whether the extracted joint points can be used according to the position and certainty of the joint points;

1.3 If the joint points can be used, divide the picture into 3 parts according to the average height of the two joint points of the shoulders and the average height of the two joint points of the hip joint, and the part above the average height of the two joint points of the shoulders is the first part. The part between the average height of the 2 joint points and the average height of the 2 joint points of the hip joint is the second part, and the part below the average height of the 2 joint points of the hip joint is the third part; if the joint point is not available, Divide the picture into three parts according to the fixed size, which are the first part, the second part and the third part respectively from top to bottom.

In the second step, the following sub-steps are specifically included:

2.1 According to the different generation parts, it is divided into three independent generation networks, corresponding to the first part, second part and third part in step 1;

2.2 For the i-th independent generative network, including the generator and the discriminator to the generator and the discriminator Input the segmented part pedestrian picture x _i , the segmented target pose mask picture p _i and the segmented target pose picture y _i , and generate a picture G _pi ( _xi , p _i ) by training the part that is consistent with the target pose;

2.3 Repeat step 2.2 for the three independent generation networks in turn to obtain all part generation pictures.

In the third step, the structured merging operation includes the following sub-steps:

3.1 For the obtained three generated part pictures corresponding to the first part, the second part and the third part respectively, according to the size ratio h _{T , i} and w _T of different parts in the original picture, the generated part pictures Scale to get the zoomed images of the 3 generated parts

3.2 According to the positional relationship of the part structure in the original picture, the Merge vertically to generate pictures for structured merged parts;

3.3 Adjust the color and edge connection information of the image generated by the structured merged part. Δh _i is the offset adjustment of the height, and c _i is the color balance adjustment factor of the different parts of the image to obtain a more realistic structured merged image A _w .

The step 2.2 specifically includes the following sub-steps:

2.2a) Input the segmented pedestrian image x _i into the generator get the generated graph Input the part pedestrian picture x _i and the target pose mask picture p _i into the generator generate image

2.2b) Input the part pedestrian picture x _i and the target pose picture y _i into the discriminator get Input the generated graph G _pi ( _xi , p _i ) and the part target pose mask image p _i into the discriminator get

2.2c) Calculate the maskL1 loss function of the part target pose picture y _i , the generated graph G _pi ( _xi ) and the part target pose mask picture p _i in ⊙ refers to the multiplication of elements between two matrices of the same size, ||*|| ₁ is the 1-norm; calculate the confrontation loss function V _pi of the generated image G _pi ( _xi ) and the real image, and Mask is the target pose mask Image matrix: is the mean value;

2.2d) Compute the adversarial loss function is the mean value;

2.2e) Combining the above two loss functions, the i-th independent generation network, the loss function is:

2.2f) The generator is updated by minimizing the loss function L _i

2.2g) By maximizing the adversarial loss function update discriminator

2.2k) Return to 2.2a) and continue to update until the loss function L _i is reduced to the threshold or the number of iterations reaches the requirement, and the image G _pi ( _xi , p _i ) is generated at the part that is consistent with the target pose.

Said step 4, the overall generation operation includes the following sub-steps:

4.1 Input the pedestrian image x into the generator G _w to obtain the generated graph G _w (x), and input the pedestrian image x, the target pose mask image, and the merged image A _w into the generator G _w to obtain the generated graph G _w (x, p, A _w );

4.2 Input the target pose picture y into the discriminator D _w to get D _w (y), and input the generated image G _w (x, p, A _w ) into the discriminator D _w to get D _w (G _w (x, p, A _w ) );

4.3 Calculate the maskL1 loss function M(G _w ) of the target pose picture y, the generated picture G _w (x) and the mask picture p:

⊙ refers to the multiplication of elements between two matrices of the same size, ||*|| ₁ is the 1-norm;

4.4 Compute the identity classification network as a guide:

Among them, cl refers to the identity category label of the target person. If the category label predicted by the classification network is consistent with cl, then Q _c = 1, otherwise Q _c = 0, P(G _w (x, p, A _w )) the output of the classification network Probability distributions;

4.5 Calculate the confrontation loss function V _w :

4.6 The overall generation network, the loss function L _w is:

L _w ＝V _w (D _w ，G _w )+M(G _w )+C(G _w ，cl)

4.7 Update the generator _Gw by minimizing the loss function Lw;

4.8 Update the discriminator D _w by maximizing the adversarial loss function V _w (D _w , G _w );

4.9 Return to step 4.1 and continue to update until the loss function L _w is reduced to an acceptable range or the number of iterations meets the requirements, and the generated image G _w (x, p, A _w ) is output.

In the step 1, the operation of extracting the mask is specifically:

For the input picture, use the mask detection algorithm to obtain the corresponding mask picture; wherein, the color of the detected object on the mask picture is uniformly white, and the background color is uniformly black.

In the step 3, the calculation formula of _Aw is:

Among them, h _T and w _T represent the height and width of the target picture, h _{T, i} represent the height of the i-th body part of the target picture; R(pic, h, w) represents resizing a picture to h*w The operation, O(h*w) refers to the zero matrix of h*w size. We reorganize the position of part images according to the part structure relations of target images. In order to ensure the smoothness of the part connection, Δh _i is the height offset adjustment, and _ci is the color balance adjustment factor of different parts of the picture.

A pedestrian image generation device based on a deformable structure, comprising:

Image preprocessing module: For the input original pedestrian picture and target pose picture, the original pedestrian picture and target pose picture are respectively segmented and extracted according to the part structure, and three sets of preprocessed part pedestrian mask pictures and part targets are obtained. Attitude mask pictures, part pedestrian pictures and part target pose pictures;

Part generation module: preprocess the part pedestrian image obtained by segmentation with the part pedestrian mask image, perform part generation operation on the part target pose mask picture, part pedestrian picture and part target pose picture, and obtain three part generated pictures;

Structured merging module: perform a structured merging operation on the three partially generated pictures obtained by the part generating operation to obtain a structured merging picture;

Overall generation module: take the structured merged image, the original image and the target pose as input, perform the overall generation operation, and obtain a final generated image of pedestrians.

Both the part generation module and the overall generation module contain a generator and a discriminator.

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

By decomposing the complex problem of pedestrian image generation into the pose conversion problem of several parts of the picture, the number of training samples required by the generation network is reduced, and more local features are used as the index of generation, which improves the generation of images while being efficient. the quality of. Specifically, it includes: through the segmentation operation and the extraction mask operation, the prior processing of the picture in line with the characteristics of the human body; through the part generation operation, different parts of the picture are generated, and the complex body posture correspondence is decomposed; through the structured merging operation, the The generated part images are combined to provide powerful guidance for the generation of the whole body; through the overall generation operation, a more authentic and credible pedestrian image is generated under the premise of retaining local information and identity information. To sum up, the method provided by the embodiment of the present invention can improve the efficiency and authenticity of the pedestrian image generation algorithm.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

Fig. 1 is a flow chart of a pedestrian image generation method based on a deformable structure in the present invention;

Fig. 2 is a comparison diagram of a pedestrian image generation method based on a deformable structure in the present invention;

3 is an overall schematic diagram of a pedestrian image generation method based on a deformable structure in an embodiment of the present invention;

FIG. 4 is a schematic diagram of segmentation and extraction operations based on deformable structure-based pedestrian image generation in an embodiment of the present invention;

FIG. 5 is a schematic diagram of a structured merging operation based on deformable structure-based pedestrian image generation in an embodiment of the present invention;

FIG. 6 is a structural block diagram of a pedestrian image generation device based on a deformable structure according to the present invention.

Detailed ways

Embodiment one

As shown in Figures 1 and 2, the present invention provides a pedestrian image generation method based on a deformable structure, which can obtain an optimized target posture picture, specifically including the following steps:

Step 1, as shown in Figure 4, for the input pedestrian picture and the target pose picture, the pedestrian picture and the target pose picture are segmented according to the part structure, the obtained part pedestrian picture and the part target pose picture, the pedestrian picture, the target pose picture The picture, the part pedestrian picture and the part target pose picture are all subjected to the mask operation to obtain the pedestrian mask picture, the target pose mask picture, the part pedestrian mask picture and the part target pose picture mask picture;

The segmentation operation specifically includes the following steps:

1.1 For pedestrian pictures and target pose pictures, use the joint point detection algorithm, first find 14 joint points of the input picture;

1.2 Judging whether the extracted joint points can be used based on the position and certainty of the joint points, the requirements for use are: the number of joint points with a certainty degree greater than 0.6 exceeds 8, and the minimum longitudinal distance between the shoulder joint point and the hip joint point More than 1/3 of the total height of the picture;

1.3 If the joint points can be used, divide the picture into 3 parts according to the average height of the two joint points of the shoulders and the average height of the two joint points of the hip joint, and the part above the average height of the two joint points of the shoulders is the first part. The part between the average height of the 2 joint points and the average height of the 2 joint points of the hip joint is the second part, and the part below the average height of the 2 joint points of the hip joint is the third part; if the joint point is not available, Divide the picture into 3 parts according to the fixed size. The specific size is: divide the picture into three parts vertically and sequentially. The height of the first part (head) accounts for 1/4 of the total height of the picture, and the height of the second part (upper body part) Accounting for 3/8 of the total height of the picture, the third part (lower body part) accounts for 3/8 of the total height of the picture.

The operation of extracting the mask is as follows:

For the input picture, use the mask detection algorithm to obtain the corresponding mask picture;

Unify the detected object color on the mask image to white, and unify the background color to black, and output the final mask image as a mask image.

Step 2, preprocessing the part pedestrian picture, performing a part generation operation on the part pedestrian picture, part target pose picture and part target pose mask picture obtained after the preprocessing, to obtain the part generation picture;

The preprocessing is to multiply the pedestrian mask image of the part by the pedestrian image of the original part to obtain the pedestrian image of the part with the background removed;

Part generation operation, the input requirements are a segmented part pedestrian picture, the mask picture corresponding to the segmented part target pose, and the segmented part target pose picture, and the output is a part generation picture consistent with the target pose; specifically includes Follow the steps below:

2.1 According to the different generation parts, it is divided into three independent generation networks, corresponding to the first part, second part and third part in step 1;

2.2 For the i-th independent generative network, including a generator and a discriminator Input a segmented pedestrian image x _i , a mask image p _i corresponding to the segmented target pose image, and a segmented target pose image y _i , and generate a picture G _pi for a part consistent with the target pose through training (x _i , p _i );

2.3 Repeat step 2.2 for the three independent generation networks in turn to obtain all the generated pictures of the parts;

The step 2.2 specifically includes the following sub-steps:

2.2a) Input the segmented pedestrian image x _i into the generator Get the generated graph G _pi ( _xi ), and input the part pedestrian image x _i and the target pose mask image p _i into the generator generate image

2.2b) Input the part pedestrian picture x _i and the target pose picture y _i into the discriminator get Input the generated graph G _pi ( _xi , p _i ) and the part target pose mask image p _i into the discriminator get

2.2c) Calculate the maskL1 loss function of the part target pose picture y _i , the generated graph G _pi ( _xi ) and the part target pose mask picture p _i in ⊙ refers to the multiplication of elements between two matrices of the same size, ||*|| ₁ is the 1-norm; calculate the confrontation loss function V _pi of the generated image G _pi ( _xi ) and the real image, and Mask is the target pose mask Image matrix: is the mean value;

2.2d) Compute the adversarial loss function is the mean value;

2.2e) Combining the above two loss functions, the i-th independent generation network, the loss function is:

2.2f) The generator is updated by minimizing the loss function L _i

2.2g) By maximizing the adversarial loss function update discriminator

2.2k) Return to 2.2a) and continue to update until the loss function L _i is reduced to the threshold or the number of iterations meets the requirement, and the output part generates a picture G _pi ( _xi , p _i ).

Step 3, performing a structured merging operation on the part generated pictures obtained by the part generating operation;

A structured merge operation includes the following sub-steps:

3.1 For the obtained three generated part pictures corresponding to the first part, the second part and the third part respectively, according to the size ratio h _{T , i} and w _T of different parts in the original picture, the generated part pictures Scale to get the image of the generated part

3.2 According to the positional relationship of the part structure in the original image, the three generated part pictures after scaling Merge vertically into a picture, that is, generate a picture of the structurally merged parts;

3.3 Adjust the information such as the color and edge connection of the image generated by the part after the structured merger, Δh _i is the offset adjustment of the height, which is obtained through multiple attempts, c _i is the color balance adjustment factor of the different parts of the picture, preferably, it can be Divide the color average of the three images by the total average of the three colors to obtain a more realistic structured merged image A _w ;

The acquisition of A _w can be obtained by the following formula:

Among them, h _T and w _T represent the height and width of the target picture, h _{T, i} represent the height of the i-th body part of the target picture; R(pic, h, w) represents resizing a picture to h*w The operation, O(h*w) refers to the zero matrix of h*w size. We reorganize the position of part images according to the part structure relations of target images. In order to ensure the smoothness of the part connection, Δh _i is the offset adjustment of the height, and c _i is the color balance adjustment factor of the pictures of different parts, as shown in Figure 5;

Step 4. Preprocess the original image, and use the merged image, pedestrian image and target pose image as input to perform an overall generation operation;

The preprocessing is to multiply the pedestrian mask image by the original pedestrian image to obtain the pedestrian image with the background removed;

The input requirements for the overall generation operation are: an original image, a target pose mask image, a target pose image, and a structurally merged part generated image; including the following sub-steps:

4.1 Input the pedestrian image x into the generator Gw to obtain the generated graph Gw(x), and input the pedestrian image x, the target pose mask image, and the merged image _Aw into the generator Gw to obtain the generated _{graph Gw (x, p, A w ) ;}

4.2 Input the target pose picture y into the discriminator D _w to get D _w (y), and input the generated graph G _w (x, p, A _w ) into the discriminator D _w to get D _w (G _w (x, p, A _w ) );

4.3 Calculate the maskL1 loss function M(G _w ) of the target pose picture y, the generated picture G _w (x) and the mask picture p:

⊙ refers to the multiplication of elements between two matrices of the same size, ||*|| ₁ is the 1-norm;

4.4 Compute the identity classification network as a guide:

Among them, cl refers to the identity category label of the target person. If the category label predicted by the classification network is consistent with cl, then Q _c = 1, otherwise Q _c = 0, P(G _w (x, p, A _w )) the output of the classification network Probability distributions;

4.5 Calculate the confrontation loss function V _w :

4.6 The overall generation network, the loss function L _w is:

L _w ＝V _w (D _w ，G _w )+M(G _w )+C(G _w ，cl)

4.7 Update the generator _Gw by minimizing the loss function Lw;

4.8 Update the discriminator D _w by maximizing the adversarial loss function V _w (D _w , G _w );

4.9 Return to step 4.1 and continue to update until the loss function L _w is reduced to an acceptable range or the number of iterations meets the requirements, and the generated image G _w (x, p, A _w ) is output.

Embodiment two

As shown in Figure 3: a pedestrian image generation method based on a deformable structure of the present invention, specifically includes the following steps:

Step 1. Through the segmentation operation and mask extraction operation, the image is subjected to a priori processing that conforms to the characteristics of the human body;

Step 2. Through the part generation operation, generate different part pictures, and decompose the complex body posture correspondence;

Step 3: Combining the generated part images through structured merging operations to provide powerful guidance for the generation of the whole body;

Step 4. Through the overall generation operation, a more authentic and credible pedestrian image is generated under the premise of retaining local information and identity information.

The invention proposes a pedestrian image generation method based on a deformable structure, which decomposes the complex problem of pedestrian image generation into the pose conversion problem of several part pictures, and solves the problem of pedestrian image generation with the idea of divide and conquer. The invention reduces the requirement of the generation network for the number of training samples, and at the same time uses more local features as the generation index, thereby improving the quality of generated pictures while being efficient. The following is a detailed description of the structured merging operation of pedestrian image generation based on deformable structures in the embodiment of the present invention.

As shown in FIG. 5, it is an exemplary flow chart of the structured merging operation based on the pedestrian image generation of the deformable structure in the embodiment of the present invention in step 3,

3.1 For the three generated part pictures obtained by the part generation operation, the generated part pictures are scaled according to the size ratio of different parts in the original image;

3.2 According to the positional relationship of the part structure in the original image, vertically merge the zoomed images of the three generated parts into one picture;

3.3 According to the smooth connection of the edges of the merged picture, fine-tune the positions of the three generated parts of the merged picture to form a smoother overall picture; adjust the three generated parts of the merged picture according to the overall color and lighting conditions of the merged picture The color and brightness weight of the picture are combined into a picture with balanced color.

As shown in Figure 6, a pedestrian image generation device based on a deformable structure of the present invention includes:

Image preprocessing module: For the input original pedestrian picture and target pose picture, the original pedestrian picture and target pose picture are respectively segmented and extracted according to the part structure, and three sets of preprocessed part pedestrian mask pictures and part targets are obtained. Attitude mask pictures, part pedestrian pictures and part target pose pictures;

Part generation module: preprocess the part pedestrian image obtained by segmentation with the part pedestrian mask image, perform part generation operation on the part target pose mask picture, part pedestrian picture and part target pose picture, and obtain three part generated pictures;

Structured merging module: perform a structured merging operation on the three partially generated pictures obtained by the part generating operation to obtain a structured merging picture;

Overall generation module: take the structured merged image, the original image and the target pose as input, perform the overall generation operation, and obtain a final generated image of pedestrians.

First, input the original pedestrian picture and target pose picture to the image preprocessing module, and perform prior processing on the picture in line with human characteristics through segmentation operation and extraction mask operation, and obtain three sets of preprocessed part pedestrian mask map and part target pose mask Pictures, part pedestrian pictures and part target pose pictures; for each group of part pedestrian pictures and part target pose pictures, through the input to the part generation module, the part generation operation is performed, different part pictures are generated, and the complex body pose correspondence is decomposed to obtain Three pictures of different parts are generated; for the three parts generated pictures obtained by the part generation operation, the merge operation is carried out by inputting the structured merging module, and the generated part pictures are combined to provide powerful guidance for the generation of the whole body. A structured merged image; the structured merged image, the original image and the target pose are used as the input of the overall generation module, and through the overall generation operation, on the premise of retaining local information and identity information, a more authentic and credible pedestrian image is finally generated .

Preferably, both the part generation module and the overall generation module include a generator and a discriminator.

Builders include:

An input processing structure that superimposes multiple input images in the third dimension;

An encoder composed of multiple convolutional layers connected in series;

A decoder consisting of multiple deconvolutional layers connected in series;

A U-shaped structure composed of the direct connection of the corresponding hierarchical network of the encoder and the decoder;

Output the output structure of the generated image and the generated loss.

Discriminators include:

The input processing structure of the picture to be discriminated and the expected label;

A feature extraction network consisting of several convolutional layers and fully connected layers;

An output structure that outputs discriminative label results and a discriminative loss.

The loss function of the generator of the part generation module is:

in,

in, is the mean value, is the mean value, y _i is the target pose picture of the part, p _i is the target pose mask picture of the part, Input the generator for the part pedestrian image x _i The resulting graph is obtained, Input the generator for the part pedestrian picture x _i and the target pose mask picture p _i The resulting graph is obtained, Input the discriminator for the part pedestrian picture x _i and the target pose picture y _i The obtained judgment result, for generating graph Input the discriminator with the position target pose mask picture p _i The obtained discrimination result, ⊙ refers to the element multiplication between two matrices of the same size, ||*|| ₁ is the 1-norm, and i represents the part, which corresponds to the three parts of the segmentation.

The discriminant function of the discriminator module of the part generation module is:

in, is the mean value, is the mean value, y _i is the target pose picture of the part, p _i is the target pose mask picture of the part, Input the generator for the part pedestrian picture x _i and the target pose mask picture p _i The resulting graph is obtained, Input the discriminator for the part pedestrian picture x _i and the target pose picture y _i The obtained judgment result, for generating graph Input the discriminator with the position target pose mask picture p _i In the obtained discrimination result, i represents a part, which specifically corresponds to the three parts of the segmentation.

The loss function of the generator G _w of the overall generation module is

L _w ＝V _w (D _w ，G _w )+M(G _w )+C(G _w ，cl)

in,

Among them, x is a pedestrian picture, y is a target pose picture, p is a target pose mask picture, A _w is a composite photo, G _w (x) is a pedestrian picture x input generator G _w to get a generated map, G _w (x, p, A _w ) is the pedestrian picture x, the target pose mask picture p, and the merged picture A _w input to the generator G _w to obtain the generated image; D _w (y) is the discriminant result obtained by inputting the target pose picture y into the discriminator D _w , D _w ( Gw(x, p, A _w )) is the discrimination result obtained by inputting the generated graph G _w (x, p, A _w ) into the discriminator D _w , Both represent the corresponding mean value, ⊙ refers to the element multiplication between two matrices of the same size, ||*|| ₁ is the 1-norm, Mask is the matrix corresponding to the target pose mask image p, and cl refers to the identity category label of the target person , if the category label predicted by the classification network is consistent with cl, Q _c =1, otherwise Q _c =0, P(G _w (x, p, A _w )) the output probability distribution of the classification network;

The discriminant function of the discriminator G _w of the overall generation module is:

Among them, x is a pedestrian picture, y is a target pose picture, p is a target pose mask picture, A _w is a composite photo, G _w (x, p, A _w ) is a pedestrian picture x, a target pose mask picture p, and a merged picture A _w Input the generator G _w to get the generated graph; D _w (y) is the discrimination result obtained by inputting the target pose image y into the discriminator D _w , and D _w (Gw(x, p, A _w )) is the generated graph G _w (x, p, A _w ) input the discriminant result obtained by the discriminator D _w , Both represent the corresponding mean.

The above description is only a preferred embodiment of the present application and an illustration of the applied technical principle. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover the technical solutions formed by the above-mentioned technical features or without departing from the above-mentioned inventive concept. Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features with similar functions disclosed in (but not limited to) this application.

Claims (10)

1. A pedestrian image generation method based on deformable structure, is characterized in that, specifically comprises the following steps: Step 1. For the input pedestrian pictures and target pose pictures, the pedestrian pictures and target pose pictures are segmented according to the part structure, and the obtained part pedestrian pictures and part target pose pictures are divided into pedestrian pictures, target pose pictures, part pedestrian pictures and The target pose pictures of the parts are all subjected to the mask operation to obtain the pedestrian mask picture, the target pose picture mask picture, the part pedestrian mask picture and the part target pose picture mask picture; Step 2, preprocessing the part pedestrian picture, performing a part generation operation on the preprocessed part pedestrian picture, part target pose picture and part target pose mask picture, to obtain the part generation picture; Step 3, performing a structured merging operation on the part generated picture obtained by the part generating operation in step 2, to obtain a structured merging picture; Step 4: Perform preprocessing on the original pedestrian picture, use the preprocessed pedestrian picture, the merged picture in step 3, and the target pose picture as input, and perform an overall generation operation to obtain a generated picture. 2. a kind of pedestrian image generation method based on deformable structure as claimed in claim 1, is characterized in that, in described step 1, segmentation operation specifically comprises the following steps: 1.1 For pedestrian pictures and target pose pictures, use the joint point detection algorithm to find the joint points of the input picture; 1.2 Judging whether the extracted joint points can be used according to the position and certainty of the joint points; 1.3 If the joint points can be used, divide the picture into 3 parts according to the average height of the two joint points of the shoulders and the average height of the two joint points of the hip joint, and the part above the average height of the two joint points of the shoulders is the first part. The part between the average height of the 2 joint points and the average height of the 2 joint points of the hip joint is the second part, and the part below the average height of the 2 joint points of the hip joint is the third part; if the joint point is not available, Divide the picture into three parts according to the fixed size, which are the first part, the second part and the third part respectively from top to bottom. 3. a kind of pedestrian image generation method based on deformable structure as claimed in claim 2, is characterized in that, in described step 2, specifically comprises the following sub-steps: 2.1 According to the different generation parts, it is divided into three independent generation networks, corresponding to the first part, second part and third part in step 1; 2.2 For the i-th independent generative network, including the generator and the discriminator to the generator and the discriminator Input the segmented part pedestrian picture x _i , the segmented target pose mask picture p _i and the segmented target pose picture y _i , and generate a picture G _pi ( _xi , p _i ) by training the part that is consistent with the target pose; 2.3 Repeat step 2.2 for the three independent generation networks in turn to obtain all part generation pictures. 4. a kind of pedestrian image generation method based on deformable structure as claimed in claim 3, is characterized in that, described step 3, structured merging operation comprises the following sub-steps: 3.1 For the obtained three generated part pictures corresponding to the first part, the second part and the third part respectively, according to the size ratio h _{T , i} and w _T of different parts in the original picture, the generated part pictures Scale to get the zoomed images of the 3 generated parts 3.2 According to the positional relationship of the part structure in the original picture, the Merge vertically to generate pictures for structured merged parts; 3.3 Adjust the color and edge connection information of the pictures generated by the structured merged parts. Δh _i is the height offset adjustment, and c _i is the color balance adjustment factor of different parts of the picture. According to Δh _i and c _i , a more realistic structure can be obtained Merge image A _w . 5. A kind of pedestrian image generation method based on deformable structure as claimed in claim 4, is characterized in that, described step 2.2 specifically comprises the following sub-steps: 2.2a) Input the segmented pedestrian image x _i into the generator Get the generated graph G _pi ( _xi ), and input the part pedestrian image x _i and the target pose mask image p _i into the generator generate image 2.2b) Input the part pedestrian picture x _i and the target pose picture y _i into the discriminator get Input the generated graph G _pi ( _xi , p _i ) and the part target pose mask image p _i into the discriminator get 2.2c) Calculate the maskL1 loss function of the part target pose picture y _i , the generated graph G _pi ( _xi ) and the part target pose mask picture p _i in ⊙ refers to the multiplication of elements between two matrices of the same size, ||*|| ₁ is the 1-norm; calculate the confrontation loss function V _pi of the generated image G _pi ( _xi ) and the real image, and Mask is the target pose mask Image matrix: is the mean value; 2.2d) Compute the adversarial loss function is the mean value; 2.2e) Combining the above two loss functions, the i-th independent generation network, the loss function is: 2.2f) The generator is updated by minimizing the loss function L _i 2.2g) By maximizing the adversarial loss function update discriminator 2.2k) Return to 2.2a) and continue to update until the loss function L _i is reduced to the threshold or the number of iterations reaches the requirement, and the image G _pi ( _xi , p _i ) is generated at the part that is consistent with the target pose. 6. a kind of pedestrian image generation method based on deformable structure as claimed in claim 5, is characterized in that, described step 4, overall generation operation comprises the following sub-steps: 4.1 Input the pedestrian image x into the generator G _w to obtain the generated graph G _w (x), and input the pedestrian image x, the target pose mask image, and the merged image A _w into the generator G _w to obtain the generated graph G _w (x, p, A _w ); 4.2 Input the target pose picture y into the discriminator D _w to get D _w (y), and input the generated image G _w (x, p, A _w ) into the discriminator D _w to get D _w (G _w (x, p, A _w ) ); 4.3 Calculate the maskL1 loss function M(G _w ) of the target pose picture y, the generated picture G _w (x) and the mask picture p: ⊙ refers to the multiplication of elements between two matrices of the same size, ||*|| ₁ is the 1-norm; 4.4 Compute the identity classification network as a guide: Among them, cl refers to the identity category label of the target person. If the category label predicted by the classification network is consistent with cl, then Q _c = 1, otherwise Q _c = 0, P(G _w (x, p, A _w )) the output of the classification network Probability distributions; 4.5 Calculate the confrontation loss function V _w : 4.6 The overall generation network, the loss function L _w is: L _w ＝V _w (D _w ，G _w )+M(G _w )+C(G _w ，cl) 4.7 Update the generator _Gw by minimizing the loss function _Lw ; 4.8 Update the discriminator D _w by maximizing the adversarial loss function V _w (D _w , G _w ); 4.9 Return to step 4.1 and continue updating until the loss function L _w is reduced to an acceptable range or the number of iterations meets the requirements, and the generated image G _w (x, p, A _w ) is output. 7. a kind of pedestrian image generation method based on deformable structure as claimed in claim 6, is characterized in that, in described step 1, extracting mask operation is specifically: For the input picture, use the mask detection algorithm to obtain the corresponding mask picture; wherein, the color of the detected object on the mask picture is uniformly white, and the background color is uniformly black. 8. a kind of pedestrian image generation method based on deformable structure as claimed in claim 4, is characterized in that, in described step 3, the computing formula of _Aw is: Among them, h _T and w _T represent the height and width of the target picture, h _{T, i} represent the height of the i-th body part of the target picture; R(pic, h, w) represents resizing a picture to h*w The operation, O(h*w) refers to the zero matrix of h*w size. We reorganize the location of part images according to the part structure relations of target images. In order to ensure the smoothness of the part connection, Δh _i is the height offset adjustment, and _ci is the color balance adjustment factor of different parts of the picture. 9. A pedestrian image generation device based on a deformable structure, characterized in that it comprises: Image preprocessing module: For the input original pedestrian picture and target pose picture, the original pedestrian picture and target pose picture are respectively segmented and extracted according to the part structure, and three sets of preprocessed part pedestrian mask pictures and part targets are obtained. Attitude mask pictures, part pedestrian pictures and part target pose pictures; Part generation module: preprocess the part pedestrian image obtained by segmentation with the part pedestrian mask image, perform part generation operation on the part target pose mask picture, part pedestrian picture and part target pose picture, and obtain three part generated pictures; Structured merging module: perform a structured merging operation on the three partially generated pictures obtained by the part generating operation to obtain a structured merging picture; Overall generation module: take the structured merged image, the original image and the target pose as input, perform the overall generation operation, and obtain a final generated image of pedestrians. 10. A pedestrian image generation device based on a deformable structure as claimed in claim 9, wherein both the part generation module and the overall generation module include a generator and a discriminator.