CN115082295B - An image editing method and device based on self-attention mechanism - Google Patents

Abstract

The invention discloses a fashion image editing method and device based on a self-attention mechanism, wherein the method comprises the following steps: extracting features of image editing information by using a cyclic convolutional neural network, rendering and refining coarse image editing results of different levels, generating refined image editing results of different levels, and predicting masks corresponding to target images; respectively extracting the refined image editing result and the characteristics of the image editing information through an encoder, and respectively traversing the selected channel image block and the spatial image block from the refined image editing result and the characteristics of the image editing information to calculate the attention weight matrix of the current level; multiplying the attention weight matrix and the feature points of the image editing information obtained from the previous level to generate features of the image editing information of the current level, and decoding the features of the image editing information through a convolutional neural network until a final fashion editing image is generated. The device comprises: a processor and a memory. The invention improves the quality and accuracy of the generated image.

Description

An image editing method and device based on self-attention mechanism

Technical Field

The present invention relates to the field of image generation in computer vision, and in particular to an image editing method and device based on a self-attention mechanism.

Background technique

With the rapid development and increasing popularity of the Internet, fashion image editing technology has been widely used in many fields. For example, virtual fitting can not only enhance consumers' experience and change traditional shopping methods, but also help reduce sales costs ^[1] . The posture-guided human image generation technology has numerous potential applications in film production, online shopping, and pedestrian re-identification. Face editing and fashion editing help inject new vitality into the fashion field and improve users' consumption experience. Using deep learning technology to generate realistic fashion images will play a more important role in fashion design and marketing, and the development of industrial intelligence.

In recent years, many research works have focused on extracting global image feature information from original images and image transformation information using convolutional neural networks, and realizing image deformation or editing by estimating the mapping relationship between features, such as APS ^[2] . However, since convolutional neural networks can only focus on information near the convolution kernel in each operation and cannot integrate information far from the convolution kernel, in the fashion image generation task, it is often necessary to consider not only the connection and influence between global information, but also the relationship between channel information and image information. Therefore, when editing an image, the original information is often destroyed.

In addition, previous work often uses thin plate spline transformation or appearance flow transformation to estimate the mapping relationship between features. For example: clothwarp ^[3] , but the thin plate spline transformation cannot accurately handle large geometric deformations, and due to the high degree of freedom and lack of appropriate regularization, the appearance flow transformation often causes severe deformation during the image transformation process, resulting in significant distortion. Texture artifacts. In addition, traditional appearance flow transformation and thin spline transformation cannot generate information that does not exist in the original image, making it impossible to effectively complete the image editing task.

Contents of the invention

The present invention provides an image editing method and device based on a self-attention mechanism. The present invention uses original images and image editing information as input data, respectively extracts high-level feature information of the two, and based on the transformation between high-level feature information and The mapping relationship estimates the multi-level appearance flow transformation matrix, uses the multi-level appearance flow transformation matrix to generate a series of rough target edited images, and uses the self-attention mechanism to capture the connection between each local information. Edit images are optimized to produce the final stylish edited image and improve the quality and accuracy of the generated image, as described below:

In the first aspect, an image editing method based on a self-attention mechanism, the method includes:

For the original image and image editing information, a convolutional neural network is used to extract the feature information of both and generate multi-layer feature information pairs;

Generate a multi-level appearance flow transformation matrix by estimating the transformation and mapping relationship between pairs of feature information, and use the appearance flow transformation matrix to transform or bend original images of different sizes to generate a series of rough image editing results of different sizes;

Use circular convolutional neural networks to extract features of image editing information, render and refine rough image editing results at different levels, generate refined image editing results at different levels and predict the mask corresponding to the target image;

The encoder extracts the features of the refined image editing results and image editing information respectively, and then traverses and selects channel image blocks and spatial image blocks from the features of the refined image editing results and image editing information to calculate the attention of the current level. weight matrix;

The features of the image editing information at the current level are generated by multiplying the attention weight matrix and the features of the image editing information obtained from the previous level, and then decoding the features of the image editing information through the convolutional neural network until the final fashion edited image is generated. .

Wherein, the original image and image editing information are: for the virtual fitting task, the original image is a character image, and the image editing information is a picture of clothing to be changed; for the posture-guided character image editing task, the original image is a A person image, the image editing information is the target human pose; for the face editing task, the original image is a face image, and the image editing information is a semantic segmentation map edited by the user; for the fashion editing task, the original image is a person Image, image editing information is a sketch edited by the user.

Furthermore, the convolutional neural network is:

Using the ResNet-based architecture, two multi-scale feature extraction networks are constructed. Each feature extraction network extracts features from the original image and image editing information respectively. Each feature extraction network contains a downsampling operation and two residual networks. Each time The downsampling operation includes one layer of convolution, one data normalization process and one activation function. Each residual network includes two layers of convolution, two data normalization processes and two activation functions;

The two multi-scale feature extraction networks respectively generate feature matrices of three different sizes with a channel number of 256. The multi-level feature information pairs are {{c ₁ , p ₁ }, {c ₂ , p ₂ }, {c ₃ ,p ₃ }}, c _i , p _i ∈R ^H×W×C , where c _i represents the feature information of the i-th layer extracted from the original image, and p _i represents the i-th layer extracted from the image editing information feature information, H, W, and C respectively represent the height, width, and number of channels of the view features, and R is a set of real numbers.

Wherein, the appearance flow transformation matrix includes: a coordinate transformation matrix and a pixel deviation matrix. The coordinate transformation matrix rearranges the pixels in the original image and is used to bend and transform the original image; the pixel deviation matrix rearranges the pixels after coordinate transformation. pixels are compensated to generate editing information not present in the original image.

Furthermore, the appearance flow transformation estimation matrix of each layer is stacked by a FlowNetSimple network and two FlowNetCor networks, which is regarded as an encoder-decoder architecture;

The encoder part of the FlowNetSimple network stacks the original image and image editing information together in the channel dimension and uses a series of convolutional layers to extract features. It contains nine convolutional layers, six of which have a stride of 2 and a nonlinear ReLU activation function after each layer.

The encoder part of the FlowNetCor network first extracts the features of the original image and image editing information through three convolutional layers, and then traverses the image blocks in the two features to perform correlation calculations. The correlation calculation formula for the image block with the center coordinates (x ₁ ,x ₂ ) is as follows:

Among them, f ₁ and f ₂ represent the characteristics of the original image and image editing information respectively, and k represents the size of the image block. By calculating the sum of the dot products of the two feature vectors at different positions in the current image block, the center coordinate is (x ₁ , x ₂ ) image block correlation and used for subsequent decoding;

In the process of stacking one FlowNetSimple network and two FlowNetCor networks, the appearance flow transformation estimation network replaces the 7x7 and 5x5 convolution kernels in the encoding module part with multi-layer 3x3 convolution kernels to increase the resolution of small displacements.

Among them, the attention weight matrix is:

The encoder extracts the features of the refined image editing result and image editing information, and traverses the image blocks corresponding to the two respectively. The kernel vector corresponding to a certain coordinate (x, y) is:

k(x,y)=M(f _s (x, y), f _t (x, y))

Among them, f _s and f _t represent the characteristics of the refined image editing result and the image editing information respectively, while f _s (x, y) and f _t (x, y) represent the refined image at the coordinate (x, y). Feature vector of the image editing result and image editing information; M represents the fully connected layer, using the softmax layer as the activation function, and outputs a one-dimensional vector representing the importance of each point in the image block at the current coordinates, that is, the kernel vector, which combines all coordinates The kernel vectors are spliced to obtain the current attention weight matrix;

The attention weight matrix and the features of the image editing information obtained from the previous level are dot multiplied and average pooled to generate the features of the image editing information of the current level for subsequent decoding.

The second aspect is an image editing device based on a self-attention mechanism. The device includes: a processor and a memory. Program instructions are stored in the memory. The processor calls the program instructions stored in the memory to cause the device to execute. The method steps of any one of the first aspects.

In a third aspect, a computer-readable storage medium stores a computer program. The computer program includes program instructions. When executed by a processor, the program instructions cause the processor to execute the first aspect. The method steps described in any one of.

The beneficial effects of the technical solution provided by the present invention are:

1. The present invention finds accurate feature mapping relationships by performing feature extraction and correlation fusion on the original image and image editing information, uses self-attention to effectively capture the connection between various local information of the image, calculates the attention weight matrix and uses The image editing information is constrained as a priori information; different from the traditional self-attention that only uses a single image block, the present invention uses two independent image blocks to connect the channel attention and spatial attention in series, and extracts the channel The affine transformation of features and spatial features enables the network to understand the spatial feature information and channel feature information of the image more accurately, enhances the network's ability to capture long-distance dependencies, and improves the accuracy of generated images.

2. By estimating appearance flow transformation matrices at multiple levels, the present invention retains and transmits feature mapping relationships at different scales, avoiding severe deformation of the original image; different from the traditional appearance flow transformation matrix that only has coordinate transformation, the present invention The coordinate transformation matrix and pixel compensation matrix are used to improve the richness of the image information generated by the appearance flow transformation; the rough image is rendered and refined through a recurrent neural network to optimize the quality of the generated image.

Therefore, the present invention can effectively estimate the feature mapping relationship between the original image and the image editing information, and capture the correlation between the local information of the image, thereby improving the quality and accuracy of the generated fashion edited image.

Description of drawings

Figure 1 is a flow chart of an image editing method based on the self-attention mechanism;

Figure 2 is a schematic diagram of the image editing method based on the self-attention mechanism, taking posture transformation as an example;

FIG3 is a schematic diagram of appearance flow transformation of an image editing method based on a self-attention mechanism, taking posture transformation as an example;

Figure 4 is a schematic diagram of the refinement network of the image editing method based on the self-attention mechanism, taking pose transformation as an example;

Figure 5 is a schematic diagram of the self-attention image generation network of the image editing method based on the self-attention mechanism, taking posture transformation as an example;

Figure 6 is a schematic structural diagram of an image editing device based on a self-attention mechanism.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

Example 1

An image editing method based on the self-attention mechanism, see Figure 1. The method includes the following steps:

Step 101: Enter the original image and image editing information;

Step 102: For the original image and image editing information, use a convolutional neural network to extract feature information from both and generate multi-level feature information pairs;

Step 103: Generate a multi-level appearance flow transformation matrix by estimating the transformation and mapping relationship between feature information pairs, and convert or bend original images of different sizes to generate a series of rough image editing results of different sizes;

Step 104: extracting features of the image editing information using a recurrent convolutional neural network, which is used to render and refine the rough image editing results at different levels, and predict the mask corresponding to the target image, and save and transmit the semantic mapping results of different feature levels;

Step 105: Use self-attention to capture the connection between various local information, traverse the channel image blocks and spatial image blocks to calculate the attention weight matrix of the current level, and generate the image editing result of the current level, and then use the convolutional neural network to edit it Decoded to produce the final stylish edited image.

To sum up, the embodiment of the present invention improves the quality and accuracy of fashion image editing through the above-mentioned steps 101 to 105, and meets the personalized needs in practical applications.

Example 2

The scheme in Example 1 is further introduced below in combination with specific calculation formulas and examples, as described below for details:

201: Enter the original image and image editing information;

For many different tasks of fashion image editing, the original image v and image editing information p are also different. For the virtual fitting task, the original image is a character image, and the image editing information is a picture of clothing to be changed; for For the posture-guided human image editing task, the original image is a human image, and the image editing information is the target human posture; for the face editing task, the original image is a human face image, and the image editing information is a semantic segmentation map edited by the user. ; For the fashion editing task, the original image is a character image, and the image editing information is a sketch edited by the user.

This method implements four fashion image editing tasks based on different input information by using a single model architecture.

202: For the original image and image editing information, use a convolutional neural network to extract feature information of both and generate multi-level feature information pairs;

For the input original image and image editing information, a convolutional neural network designed based on the ResNet ^[4] architecture is used to extract feature information pairs at different levels. Specifically, two multi-scale feature extraction networks are constructed. Each network extracts features from the original image and image editing information respectively. Each feature extraction network contains one downsampling operation and two residual networks. Each downsampling operation It contains one layer of convolution, one data normalization process and one activation function. Each residual network contains two layers of convolution, two data normalization processes and two activation functions. The convolution kernel size is 3×3, and the activation function selects the ReLU activation function. The ReLU function is ReLU(x)=max(x,0), where max is the maximum value function. The two multi-scale feature extraction networks generate feature matrices of three different sizes with a channel number of 256, forming multi-level feature information pairs. The obtained multi-level feature information pairs are {{c ₁ , p ₁ }, {c ₂ , p ₂ }, {c ₃ , p ₃ }}, c _i , p _i ∈R ^H×W×C , where c _i represents the feature information of the i-th layer extracted from the original image, p _i represents the feature information of the i-th layer extracted from the image editing information, H, W, and C respectively represent the height, width, and channel number of the view feature, R is the set of real numbers.

203: Generate a multi-level appearance flow transformation matrix by estimating the transformation and mapping relationship between feature information pairs, and convert or bend original images of different sizes to generate a series of rough image editing results of different sizes; for each In terms of level, the feature information pair and the appearance flow transformation matrix generated by the previous level are used as input, and the redesigned appearance flow transformation estimation network is used to calculate the appearance flow transformation matrix of the current level. Take the second layer as an example:

f ₂ =F({c ₂ ,p ₂ },f ₁ ) (1)

where {c ₂ , p ₂ } is the feature information pair of the second layer, f ₁ represents the calculated appearance flow transformation matrix of the first layer, f ₂ represents the calculated appearance flow transformation matrix of the second layer, and F represents the current Layers of appearance flow transform estimation networks.

Specifically, each layer of appearance flow transformation estimation network is stacked by one FlowNetSimple ^[5] network and two FlowNetCorr ^[5] networks. Compared with a single FlowNetSimple network or a single FlowNetCorr network, the stacked network can Effectively prevent overfitting. Both the FlowNetSimple network and the FlowNetCorr network can be regarded as an encoder-decoder architecture. The encoder part of the FlowNetSimple network stacks the original image and image editing information together by channel dimension, and then uses a series of convolutional layers to extract features, including nine convolutional layers, six of which have a stride of 2. There is also a nonlinear ReLU activation function behind each layer. The encoder part of the FlowNetCor network first extracts the features of the original image and the image editing information through three convolutional layers, and then traverses the image blocks in the two features to perform correlation calculations. The image with the center coordinates (x ₁ , x ₂ ) The correlation formula of the block is as follows:

where f ₁ and f ₂ represent the characteristics of the original image and image editing information respectively, and k represents the size of the image block. By calculating the sum of the dot products of the two feature vectors at different positions in the current image block, the center coordinate is (x ₁ ,x ₂ ) of the image block and used for subsequent decoding. The appearance flow transformation estimation network of the embodiment of the present invention is also improved for small displacement situations. The convolution kernels with sizes of 7x7 and 5x5 in the encoding module part are replaced with multi-layer 3x3 convolution kernels to increase the resolution of small displacements. .

Use the appearance flow transformation matrix generated by each stage To original pictures of different sizes/> Perform transformations to produce a series of rough image edits of varying sizes/> used for subsequent image generation. Different from the original appearance flow transformation matrix based on linear coordinate transformation, the appearance flow transformation matrix in the embodiment of the present invention not only includes a coordinate transformation matrix, but also includes a pixel deviation matrix. In fashion editing tasks, it is often necessary to generate information that has never appeared in the original image. If you only rely on coordinate transformation to generate rough image editing images, it will easily cause serious distortion of the image. Through the pixel deviation matrix, pixels can be compensated after coordinate transformation to generate editing information that is not in the original image.

204: Use circular convolutional neural network to extract features of image editing information, which is used to render and refine rough image editing results at different levels, predict the mask corresponding to the target image, and save and transfer the semantic mapping of different feature levels. result;

Edit the result of the generated rough image The embodiment of the present invention uses a cyclic residual convolutional neural network R2U_Ne ^t[6] to render a rough image. Compared with a convolutional neural network, a cyclic residual convolutional neural network uses a residual block instead of a traditional convolutional layer plus an activation function during the encoding and decoding process, which can effectively increase the network depth; a cyclic residual convolutional layer is used for feature accumulation, which is helpful for feature extraction. During the rendering process, the image editing information is used as a guide to generate a series of rendered and edited images at different levels/> and their respective corresponding masks/> Right now:

u _i ,m _i =R( _wi ,p) (2)

Eliminate excess information from the rendered image and retain necessary information from the original image through the resulting mask:

Among them, ⊙ represents element-by-element multiplication, represents the image obtained after sampling the original image, and _vi represents the generated refined target edited image.

205: Use self-attention to capture the connection between various local information, traverse the image blocks to generate the image editing result of the current level, and then decode it through the convolutional neural network to generate the final fashion edited image.

Among them, the self-attention mechanism takes the features of the refined image, image editing information and the mask of the corresponding level as input, and performs feature correlation to calculate the attention weight matrix of the image to capture the key information in the image editing information. Specifically:

Traverse and select image blocks f _s (x, y), f _t (x, y) from the features f _s of the refined image editing result and the features f _t of the image editing information respectively, which is different from the traditional self-attention mechanism. For a single n×n image block, the embodiment of the present invention uses two independent n×n image blocks, traverses along the spatial dimension and the channel dimension, and connects the channel attention and the spatial attention in series, effectively improving each attention. The connection between weight information in channel and space.

where f _s (x, y) and f _t (x, y) represent the refined image editing result and the feature vector of the image editing information at the coordinate (x, y). M represents the fully connected layer, which uses the softmax layer as the activation function to output a one-dimensional vector representing the importance of each point in the image block at the current coordinates, that is, the kernel vector k(x,y).

k(x,y)＝M(f _s (x, y), f _t (x, y)) (4)

The attention weight matrix is generated using the kernel vector k(x,y) of all generated image blocks. The dot product operation is used to calculate the result of the self-attention mechanism for the image block at (x,y) in the feature f _t of the image editing information. The global average pooling operation is used to obtain the image editing result p(x,y) of the coordinate (x,y), that is:

p(x,y)＝Pooling(k(x,y)⊙f _t (x,y)) (5)

Compared with using f _t (x, y) as the feature of the current level of image editing information for decoding, the feature p (x, y) obtained by using the attention mechanism can allow the model to focus on the important information in the image editing information. And fully learn and absorb.

The feature p _attn of the image editing information of the current level is generated by traversing and calculating all the image blocks, and the redundant information in the generated features is eliminated through the generated mask m and the necessary information in the original image editing information is retained:

p _out = _m⊙pattn + (1-m) _⊙ft (6)

Gradually use the self-attention module and decoder on different levels to get the final image editing result.

To sum up, the embodiment of the present invention improves the quality and accuracy of fashion image editing through the above-mentioned steps 201 to 205, and meets the personalized needs in practical applications.

Example 3

An image editing device based on a self-attention mechanism. The device includes: a processor 1 and a memory 2. Program instructions are stored in the memory 2. The processor 1 calls the program instructions stored in the memory 2 to cause the device to execute Embodiment 1. The following method steps: for the original image and image editing information, use a convolutional neural network to extract feature information of both and generate multi-level feature information pairs;

Generate a multi-level appearance flow transformation matrix by estimating the transformation and mapping relationship between pairs of feature information, and use the appearance flow transformation matrix to transform or bend original images of different sizes to generate a series of rough image editing results of different sizes;

Use circular convolutional neural networks to extract features of image editing information, render and refine rough image editing results at different levels, generate refined image editing results at different levels and predict the mask corresponding to the target image;

The features of the refined image editing results and the image editing information are extracted through the encoder, and then the channel image blocks and the spatial image blocks are traversed and selected from the features of the refined image editing results and the image editing information to calculate the attention weight matrix of the current level;

The features of the image editing information at the current level are generated by multiplying the attention weight matrix and the feature points of the image editing information obtained at the previous level. The features of the image editing information are then decoded through a convolutional neural network until the final fashion editing image is generated.

Among them, the original image and image editing information are: for the virtual fitting task, the original image is a character image, and the image editing information is a picture of clothing to be changed; for the posture-guided character image editing task, the original image is a character image Image, the image editing information is the target human pose; for the face editing task, the original image is a face image, and the image editing information is a semantic segmentation map edited by the user; for the fashion editing task, the original image is a person image, The image editing information is a sketch edited by the user.

Further, the convolutional neural network is:

Using the ResNet-based architecture, two multi-scale feature extraction networks are constructed. Each feature extraction network extracts features from the original image and image editing information respectively. Each feature extraction network contains a downsampling operation and two residual networks. Each time The downsampling operation includes one layer of convolution, one data normalization process and one activation function. Each residual network includes two layers of convolution, two data normalization processes and two activation functions;

The two multi-scale feature extraction networks respectively generate feature matrices of three different sizes with a channel number of 256. The multi-level feature information pairs are {{c ₁ , p ₁ }, {c ₂ , p ₂ }, {c ₃ ,p ₃ }}, c _i , p _i ∈R ^H×W×C , where c _i represents the feature information of the i-th layer extracted from the original image, and p _i represents the i-th layer extracted from the image editing information feature information, H, W, and C respectively represent the height, width, and number of channels of the view features, and R is a set of real numbers.

Among them, the appearance flow transformation matrix includes: a coordinate transformation matrix and a pixel deviation matrix. The coordinate transformation matrix rearranges the pixels in the original image and is used to bend and transform the original image; the pixel deviation matrix compensates the pixels after coordinate transformation, using to generate editing information not present in the original image.

Furthermore, the appearance flow transformation estimation matrix of each layer is stacked by a FlowNetSimple network and two FlowNetCor networks, which is regarded as an encoder-decoder architecture;

The encoder part of the FlowNetSimple network stacks the original image and image editing information together according to the channel dimension, and uses a series of convolutional layers to extract features, including nine convolutional layers, six of which have a stride of 2, each A nonlinear ReLU activation function is also set after the layer;

The encoder part of the FlowNetCor network first extracts the features of the original image and the image editing information through three convolutional layers, and then traverses the image blocks in the two features to perform correlation calculations. The image with the center coordinates (x ₁ , x ₂ ) The correlation formula of the block is as follows:

Among them, f ₁ and f ₂ represent the characteristics of the original image and image editing information respectively, and k represents the size of the image block. By calculating the sum of the dot products of the two feature vectors at different positions in the current image block, the center coordinate is (x ₁ , x ₂ ) image block correlation and used for subsequent decoding;

In the process of stacking one FlowNetSimple network and two FlowNetCor networks, the appearance flow transformation estimation network replaces the 7x7 and 5x5 convolution kernels in the encoding module part with multi-layer 3x3 convolution kernels to increase the resolution of small displacements.

Among them, the attention weight matrix is:

The encoder extracts the features of the refined image editing result and image editing information, and traverses the image blocks corresponding to the two respectively. The kernel vector corresponding to a certain coordinate (x, y) is:

k(x,y)=M(f _s (x, y), f _t (x, y))

Among them, f _s and f _t represent the characteristics of the refined image editing result and the image editing information respectively, while f _s (x, y) and f _t (x, y) represent the refined image at the coordinate (x, y). Feature vector of the image editing result and image editing information; M represents the fully connected layer, using the softmax layer as the activation function, and outputs a one-dimensional vector representing the importance of each point in the image block at the current coordinates, that is, the kernel vector, which combines all coordinates The kernel vectors are spliced to obtain the current attention weight matrix;

The attention weight matrix and the features of the image editing information obtained from the previous level are dot multiplied and average pooled to generate the features of the image editing information of the current level for subsequent decoding.

It should be pointed out here that the device description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the present invention will not be described again here.

The execution subjects of the above-mentioned processor 1 and memory 2 can be computers, microcontrollers, microcontrollers and other devices with computing functions. During specific implementation, the embodiments of the present invention do not limit the execution subjects, and they can be selected according to the needs of actual applications. .

The data signal is transmitted between the memory 2 and the processor 1 via the bus 3, which will not be described in detail in the embodiment of the present invention.

Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium. The storage medium includes a stored program. When the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiments.

The computer-readable storage medium includes but is not limited to flash memory, hard disk, solid state drive, etc.

It should be pointed out here that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be described again here.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, processes or functions according to embodiments of the present invention are generated in whole or in part.

The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. Computer instructions may be stored in or transmitted over a computer-readable storage medium. Computer-readable storage media can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or other integrated media that contains one or more available media. Available media may be magnetic media or semiconductor media, etc.

references

[1]Ge Y,Song Y,Zhang R,et al.Parser-free virtual try-on viadistilling appearanceflows[C]//Proceedings of the IEEE/CVF Conference onComputer Vision and PatternRecognition.2021:8485-8493.

[2]Huang S, Xiong H, Cheng Z Q, et al. Generating person images with appearance-aware posestylizer[J]. arXiv preprint arXiv:2007.09077,2020.

[3]Han X, Hu

[4]He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-778.

[5]Ilg E, Mayer N, Saikia T, et al. Flownet 2.0: Evolution of optical flowestimation with deepnetworks[C]//Proceedings of the IEEE conference oncomputer vision and patternrecognition.2017:2462-2470.

[6]Alom M Z,Hasan M,Yakopcic C,et al.Recurrent residual convolutional neural networkbased on u-net(r2u-net)for medical image segmentation[J].arXivpreprintarXiv:1802.06955,2018.

The embodiments of the present invention do not limit the models of each device unless otherwise specified, as long as the devices can complete the above functions.

Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (4)

1. An image editing method based on a self-attention mechanism, characterized in that the method comprises: For the original image and image editing information, a convolutional neural network is used to extract the feature information of both and generate multi-layer feature information pairs; A multi-level appearance flow transformation matrix is generated by estimating the transformation and mapping relationship between feature information pairs, and the appearance flow transformation matrix is used to transform or warp original images of different sizes to generate a series of rough image editing results of different sizes; The recurrent convolutional neural network is used to extract the features of image editing information, and the rough image editing results at different levels are rendered and refined, and the refined image editing results at different levels are generated and the mask corresponding to the target image is predicted; The encoder extracts the features of the refined image editing results and image editing information respectively, and then traverses and selects channel image blocks and spatial image blocks from the features of the refined image editing results and image editing information to calculate the attention of the current level. weight matrix; Multiply the attention weight matrix and the features of the image editing information obtained from the previous level to generate the features of the image editing information of the current level, and then decode the features of the image editing information through the convolutional neural network until the final edited image is generated; Wherein, the original image and image editing information are: For the virtual fitting task, the original image is a character image, and the image editing information is a picture of the clothing to be changed; for the posture-guided character image editing task, the original image is a character image, and the image editing information is the target human posture; For the face editing task, the original image is a face image, and the image editing information is a semantic segmentation map edited by the user; for the fashion editing task, the original image is a character image, and the image editing information is a sketch edited by the user; The convolutional neural network is: Using the ResNet-based architecture, two multi-scale feature extraction networks are constructed. Each feature extraction network extracts features from the original image and image editing information respectively. Each feature extraction network contains a downsampling operation and two residual networks. Each time The downsampling operation includes one layer of convolution, one data normalization process and one activation function. Each residual network includes two layers of convolution, two data normalization processes and two activation functions; The two multi-scale feature extraction networks respectively generate feature matrices of three different sizes with a channel number of 256. The multi-level feature information pairs are {{c ₁ , p ₁ }, {c ₂ , p ₂ }, {c ₃ ,p ₃ }}, c _i , p _i ∈R ^H×W×C , where c _i represents the feature information of the i-th layer extracted from the original image, and p _i represents the i-th layer extracted from the image editing information feature information, H, W, and C respectively represent the height, width, and channel number of the view features, and R is a set of real numbers; The appearance flow transformation matrix includes: a coordinate transformation matrix and a pixel deviation matrix, wherein the coordinate transformation matrix rearranges the pixels in the original image to bend and transform the original image; the pixel deviation matrix compensates the pixels after the coordinate transformation to generate editing information not in the original image; The attention weight matrix is: The encoder extracts the features of the refined image editing result and image editing information, and traverses the image blocks corresponding to the two respectively. The kernel vector corresponding to a certain coordinate (x, y) is: k(x,y)=M(f _s (x, y), f _t (x, y)) Wherein, _fs and _ft represent the features of the refined image editing result and image editing information, respectively, and _fs (x,y) and _ft (x,y) represent the feature vectors of the refined image editing result and image editing information at the coordinate (x,y); M represents a fully connected layer, which uses the softmax layer as the activation function and outputs a one-dimensional vector representing the importance of each point in the image block at the current coordinate, namely, the kernel vector. The kernel vectors of all coordinates are concatenated to obtain the current attention weight matrix; The attention weight matrix and the features of the image editing information obtained from the previous level are dot multiplied and average pooled to generate the features of the image editing information of the current level for subsequent decoding. 2. According to the image editing method based on the self-attention mechanism of claim 1, it is characterized in that each layer of the appearance flow transformation estimation network is composed of a FlowNetSimple network and two FlowNetCor networks stacked together, which is regarded as an encoder-decoder architecture; The encoder part of the FlowNetSimple network stacks the original image and image editing information together according to the channel dimension, and uses a series of convolutional layers to extract features, including nine convolutional layers, six of which have a stride of 2, each A nonlinear ReLU activation function is also set after the layer; The encoder part of the FlowNetCor network first extracts the features of the original image and the image editing information through three convolutional layers, and then traverses the image blocks in the two features to perform correlation calculations. The image with the center coordinates (x ₁ , x ₂ ) The correlation formula of the block is as follows: Among them, f ₁ and f ₂ represent the characteristics of the original image and image editing information respectively, and k represents the size of the image block. By calculating the sum of the dot products of the two feature vectors at different positions in the current image block, the center coordinate is (x ₁ , x ₂ ) image block correlation and used for subsequent decoding; In the process of stacking one FlowNetSimple network and two FlowNetCor networks, the appearance flow transformation estimation network replaces the 7x7 and 5x5 convolution kernels in the encoding module part with multi-layer 3x3 convolution kernels to increase the resolution of small displacements. 3. An image editing device based on a self-attention mechanism, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute the method steps described in any one of claims 1-2. 4. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program includes program instructions, and when executed by a processor, the program instructions cause the processor to execute the right The method steps described in any one of claims 1-2.