CN116228520A - Image Compression Sensing Reconstruction Method and System Based on Transformer Generative Adversarial Network - Google Patents

Abstract

The invention belongs to the technical field of image processing, and particularly relates to an image compression perception reconstruction method and system based on Transformer generation confrontation network. The method includes obtaining an image sample set, dividing it into a training set and a test set in proportion, and preprocessing the image ;According to the sampling rate, construct the Transformer generation confrontation network model; set the hyperparameters of the Transformer generation confrontation network model, select the loss function and optimization method; use the image data set to train the network model at different sampling rates, and pass the loss function and optimization method Train and learn the optimal parameters of the network model, and obtain the trained Transformer generation confrontation network model at different sampling rates; use the trained Transformer generation confrontation network model to perform image compression perception reconstruction, and use the evaluation index to verify the performance of the network. The present invention uses the Transformer Block based on the attention mechanism to construct a deep generative confrontation network, which significantly improves the quality of the reconstructed image.

Description

Image Compression Sensing Reconstruction Method Based on Transformer Generative Adversarial Network and system

technical field

The invention belongs to the technical field of image processing, and in particular relates to an image compression perception reconstruction method and system based on a Transformer generation confrontation network.

Background technique

With the advent of the big data information age, the shortcomings of relying on traditional sampling theorem for data sampling become more and more obvious. As an advanced data sampling theory, compressed sensing is based on the compressibility of signals to realize the perception of reconstructed signals for low-dimensional measurement values. However, the traditional iterative optimized compressed sensing reconstruction algorithm has high time complexity, and the reconstruction effect is not ideal at low sampling rates. With the development of deep learning, the proposed compressed sensing model based on deep learning greatly reduces the time complexity of the reconstruction algorithm and improves the reconstruction effect.

Contents of the invention

Aiming at the problems existing in the prior art, the present invention proposes an image compression perception reconstruction method and system based on a Transformer generation confrontational network, using an attention mechanism-based Transformer Block to construct a deep generational confrontational network, which significantly improves the quality of the reconstructed image.

In order to achieve the above object, the present invention adopts the following technical solutions:

The present invention provides a kind of image compression sensing reconstruction method based on Transformer generation confrontation network, comprising the following steps:

Obtain an image sample set, divide it into a training set and a test set in proportion, and preprocess the image;

According to the sampling rate, build a Transformer to generate an adversarial network model;

Set the hyperparameters of Transformer to generate the confrontation network model, select the loss function and optimization method;

Use the image data set to train the network model at different sampling rates, train and learn the optimal parameters of the network model through loss functions and optimization methods, and obtain the trained Transformer generation confrontation network model at different sampling rates;

Use the trained Transformer to generate an adversarial network model for image compression sensing reconstruction, and use evaluation indicators to verify the performance of the network.

Further, each batchsize resizes the image to 64×64 pixels before training.

Further, the Transformer generation confrontation network model includes a sampling network, a generation network and a discrimination network.

Further, the sampling network uses a convolution layer with a convolution kernel size of 32×32 and a step size of 32 to generate measured values, and the number of output channels is set according to the sampling rate.

Further, the generating network includes a Flatten layer, a fully connected layer, a first hidden layer, and a second hidden layer, and the measurement value generated by the sampling network is first flattened into one dimension through the Flatten layer, and then expanded through the fully connected layer. Click to 24567, and then adjust the output of the fully connected layer to the specified image size through the reshape operation; the first hidden layer is the Transformer Block, where the Transformer Block contains the original image and its corresponding position code, PixelNorm normalization layer, multi-head Self-attention mechanism, PixelNorm normalization layer and MLP layer; the second hidden layer is a sub-pixel convolution block, which includes a size of 3×3 convolution layer, batch normalization layer, SELU Activation function layer, sub-pixel convolution layer and SELU activation function layer.

Further, the identification network judges whether the image generated by the generation network is real, including multiple convolutional layers, batch normalization layers, Flatten layers and fully connected layers.

Further, setting the hyperparameters of the Transformer generation confrontation network model includes: setting the initial learning rate learning rate to 0.001, and setting the number of network iterations to 20.

Further, the Adam optimization algorithm is used to train and update the parameters of the generation network, and the RMSProp optimization algorithm is used to train and update the parameters of the identification network.

Furthermore, the performance of the network is verified by using the evaluation indexes PSNR and structural similarity SSIM.

The present invention also provides an image compression sensing reconstruction system based on Transformer generation confrontation network, comprising:

The image sample set acquisition module is used to obtain the image sample set, divide it into training set and test set in proportion, and preprocess the image;

The network model construction module is used to construct a Transformer to generate an adversarial network model according to the sampling rate;

The hyperparameter setting module is used to set the hyperparameters of the Transformer generation confrontation network model, select the loss function and the optimization method;

The training module is used to use the image data set to train the network model at different sampling rates, train and learn the optimal parameters of the network model through loss functions and optimization methods, and obtain the trained Transformer generation confrontation network model at different sampling rates;

The image reconstruction module is used to use the trained Transformer generation confrontation network model to perform image compression sensing reconstruction, and use the evaluation index to verify the performance of the network.

Compared with the prior art, the present invention has the following advantages:

1. The present invention uses the convolutional layer as a sampling network to simulate the traditional compressed sensing measurement process to measure the image and obtain the measured value, which improves the correlation between the measured value and the image. Compared with the traditional compressed sensing algorithm based on block sampling, the present invention solves the problem of The problem of image block effect existing in the existing block-based sampling is solved. The measured value goes through the fully connected layer and reshape operation of the convolutional neural network to complete the preliminary reconstruction from the measured vector to the original image. Use the Transformer Block based on the attention mechanism to build a deep generative confrontation network, and the preliminary reconstruction image is sent to the deep generative confrontation network. The attention mechanism in the Transformer Block can obtain the global information of the initial reconstruction image, and ensure that the initial reconstruction image and attention The content dependence between weights increases the receptive field of the network to capture more contextual information, and iteratively improves the reconstructed image quality through generative confrontation.

2. The image compression sensing reconstruction method of the present invention can be applied in the field of medical MRI. Compared with the traditional scanning method, the imaging speed is greatly accelerated, the imaging quality is improved, and the scanning time is shortened. At the same time of low time cost, accurate and efficient images are obtained, and the detailed information of the images is preserved; while reducing the patient's scanning time, it is also conducive to the doctor's rapid diagnosis.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic flow diagram of an image compression sensing reconstruction method based on Transformer generated confrontation network according to an embodiment of the present invention;

FIG. 2 is a structural diagram of Transformer generating an adversarial network model according to an embodiment of the present invention;

3 is a structural diagram of a sampling network and a generating network according to an embodiment of the present invention;

Fig. 4 is a structural diagram of an authentication network according to an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

As shown in Figure 1, this embodiment provides an image compression sensing reconstruction method based on Transformer generation confrontation network, including the following steps:

Step S1, obtain an image sample set, divide it into a training set and a test set in proportion, and preprocess the image.

In step S2, according to the sampling rate, a Transformer generation confrontation network model is constructed.

Step S3, setting Transformer to generate hyperparameters of the confrontation network model, selecting a loss function and an optimization method.

Step S4, use the image data set to train the network model at different sampling rates, learn the optimal parameters of the network model through loss function and optimization method training, and obtain the Transformer generation confrontation network model trained at different sampling rates.

Step S5, using the trained Transformer to generate an adversarial network model to perform image compression perception reconstruction, and use the evaluation index to verify the performance of the network.

In step S1, 202,599 178×218 images of CelebA in the public training dataset are divided into 162,770 images as a training set, 19,867 images as a verification set, and 19,962 images as a test set. Before training, images are resized to 64×64 pixels per batchsize.

Further, the Transformer generation confrontation network model in step S2 includes three parts: a sampling network, a generation network and a discrimination network, as shown in FIG. 2 .

As shown in Figure 3, the sampling network uses a convolution layer with a convolution kernel size of 32×32 and a step size of 32 to generate measurement values, and the number of output channels is set according to the sampling rate. For an image with a size of M×N, the convolution layer with a convolution kernel size of B×B×l and a step size of B×B is used to simulate the sampling operation, and the size of the finally obtained measurement value is

As shown in Figure 3, the generation network includes a flatten layer, a fully connected layer, a first hidden layer and a second hidden layer. The measurement value generated by the sampling network is first flattened into one dimension through the Flatten layer, and then the node is expanded to 24567 through the fully connected layer, and then the output of the fully connected layer 24567 is reshaped into an image size of 8*8 through the reshape operation. The number of feature maps is 384.

The first hidden layer is the Transformer Block, which uses the Transformer Block to reconstruct the image with high quality. The Transformer Block contains the original image and its corresponding position code, PixelNorm normalization layer, multi-head self-attention mechanism, PixelNorm normalization layer and MLP layer. Its structure can be expressed as: [Embedded Patches-Pixel Norm-Multi headself Attention-Pixel Norm-MLP], and the number of output channels is 384. The multi-head self-attention mechanism (Multi head self-Attention) in the Transformer Block can divide the image into blocks and extract the global information of the image in the block. The attention mechanism can capture more context information and calculate the texture complexity of each part of the image. weight, and allocate computing resources according to the size of the weight. The image size is enlarged step by step with sub-pixel convolution, and the reconstruction quality is continuously improved by generating confrontation.

The second hidden layer is a sub-pixel convolution block, which multiplies the image size through the sub-pixel convolution block. The sub-pixel convolution block includes a 3×3 convolution layer, a batch normalization layer, and a SELU activation function layer. , sub-pixel convolution layer and SELU activation function layer, its structure can be expressed as [Conv _3×3 -BN-SeLU-subpixelConv _3×3 ], Transformer Block output image size is 8*8, the number of output channels is 384, after The sub-pixel convolution changes the image size to 16*16, the number of channels is 96, and then after multiple sub-pixel convolutions and Transformer Block combinations, the image size becomes 64*64, the number of channels becomes 6, and then a convolution layer, and finally obtain a reconstructed image with 3 channels and an image size of 64*64.

As shown in Figure 4, the identification network judges whether the image generated by the generation network is true. The identification network includes multiple convolutional layers, batch normalization layers, Flatten layers and fully connected layers. Its structure can be expressed as [Conv _3×3 -LreLU-BN-…-Conv _3×3 -BN-Flatten-DenseLayer], the number of channels of the input image is 3, and then the dimension is continuously increased to 512 through multiple convolutional layers, and then the number of channels continues to expand through the fully connected layer to 1024, and finally output a true/false result. Through the confrontation between the generation network and the identification network, iterative training, the identification network can help the generation network to better optimize parameters, thereby improving the quality of the reconstructed image.

In step S3, the images in the training set are input into the network model built in step S2 according to the batch size, and an appropriate batch size is set according to the hardware conditions. In this example, the batch size is set to 16, and the Transformer is set to generate the confrontation network model. Hyperparameters: Set the initial learning rate learning rate to 0.001, and set the number of network iterations to 20.

(1) Set the target loss function of the generated network as:

为生成网络输出的重建图像。利用Adam优化算法对生成网络的参数进行训练并更新。Among them, n is the number of training images in the training set, _xi is the original image,

is the reconstructed image output by the generating network. The parameters of the generated network are trained and updated using the Adam optimization algorithm.

(2) Set the target loss function of the discriminator network as:

为生成网络输出的重建图像。利用RMSProp优化算法对鉴别网络的参数进行训练并更新。Among them, n is the number of training images in the training set, _xi is the original image,

is the reconstructed image output by the generating network. The parameters of the discriminant network are trained and updated using the RMSProp optimization algorithm.

Specifically, step S4 specifically includes the following steps:

Step S401, setting the number of channels of the sampling convolution layer according to the sampling rate.

Step S402, saving the trained model in .npz format.

Specifically, in step S5, the evaluation index peak signal-to-noise ratio PSNR and structural similarity SSIM are used to verify the performance of the network, which specifically includes the following steps:

Step S501, select the images in the test set, input them into the trained Transformer generation confrontation network model, obtain the measurement value through the sampling network, and enter the generation network to finally output the reconstructed image.

Step S502, using PSNR to measure the reconstruction effect of the network model, wherein the larger the PSNR, the better the reconstruction effect, and the calculation formula is as follows:

Mean square error (MSE):

和参考图像f(i,j)的均方误差；M，N分别为图像的高度和宽度。MSE indicates the current image

and the mean square error of the reference image f(i,j); M, N are the height and width of the image, respectively.

Peak Signal-to-Noise Ratio (PSNR):

n is the number of bits per pixel, generally 8, that is, the number of gray scales of the pixel is 256, and the unit is dB.

Step S503, using SSIM to measure the reconstruction effect of the network model, wherein the larger the SSIM, the better the reconstruction effect. For a given image x and y, the SSIM calculation formula of the two images is as follows:

为x的方差，/>

为t的方差，σ_xy为x，y的协方差，c₁＝(k₁L)²,c₂＝(k₂L)²是用来维持稳定的常数，L是像素值的动态范围，k₁＝0.01，k₂＝0.03。where μ _x is the mean value of x, μ _y is the mean value of y,

is the variance of x, />

is the variance of t, σ _xy is the covariance of x and y, c ₁ =(k ₁ L) ² , c ₂ =(k ₂ L) ² is a constant used to maintain stability, L is the dynamic range of pixel values, k ₁ =0.01, k ₂ =0.03.

Compared with the sub-pixel convolution confrontational neural network SCGAN, the network model used in the present invention has an average PSNR increase of 2.0018dB and an average SSIM increase of 0.0609 on the MNIST data set. On the Fashion-MNIST dataset, PSNR has been improved by 1.0031dB on average, and SSIM has been improved by 0.0513 on average. On the CelebA dataset, PSNR is improved by 1.2301dB on average, and SSIM is improved by 0.1123 on average. Experimental results show that this method has a better reconstruction effect than the current advanced deep compressed sensing algorithm.

Corresponding to the above-mentioned image compression perception reconstruction method based on Transformer generation confrontation network, this embodiment also provides an image compression perception reconstruction system based on Transformer generation confrontation network, including image sample set acquisition module, network model construction module, super Parameter setting module, training module and image reconstruction module.

The image sample set acquisition module is used to obtain the image sample set, divide it into training set and test set in proportion, and preprocess the image;

The network model construction module is used to construct a Transformer to generate an adversarial network model according to the sampling rate;

The hyperparameter setting module is used to set the hyperparameters of the Transformer generation confrontation network model, select the loss function and the optimization method;

The training module is used to use the image data set to train the network model at different sampling rates, train and learn the optimal parameters of the network model through loss functions and optimization methods, and obtain the trained Transformer generation confrontation network model at different sampling rates;

The image reconstruction module is used to use the trained Transformer generation confrontation network model to perform image compression sensing reconstruction, and use the evaluation index to verify the performance of the network.

It should be noted that, in this document, the terms "comprising", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or apparatus.

Finally, it should be noted that the above descriptions are only preferred embodiments of the present invention, and are only used to illustrate the technical solution of the present invention, and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (10)

1. The image compressed sensing reconstruction method based on the Transformer generating countermeasure network is characterized by comprising the following steps of:

acquiring an image sample set, dividing the image sample set into a training set and a testing set according to a proportion, and preprocessing an image;

constructing a transducer to generate an countermeasure network model according to the sampling rate;

setting a transducer to generate super parameters of an antagonistic network model, and selecting a loss function and an optimization method;

training the network model by using the image data set under different sampling rates, and training the optimal parameters of the learning network model by using a loss function and an optimization method to obtain a trained Transformer under different sampling rates to generate an antagonistic network model;

image compressed sensing reconstruction is performed on the countermeasure network model by using the trained transducer generation, and the performance of the network is verified by using the evaluation index.

2. The transform-based generation of image compressed sensing reconstruction method of claim 1, wherein each batch size resizes the image to 64 x 64 pixels prior to training.

3. The method of claim 1, wherein the transform generation countermeasure network model comprises a sampling network, a generation network, and an authentication network.

4. A method of reconstructing compressed sensing of an image based on a transform generation countermeasure network according to claim 3, wherein the sampling network uses a convolution layer with a convolution kernel size of 32 x 32 and a step size of 32 to generate the measured value, and the number of output channels is set according to the sampling rate.

5. The method for reconstructing image compressed sensing based on a Transformer generating countermeasure network according to claim 4, wherein the generating network comprises a flat layer, a full-connection layer, a first hidden layer and a second hidden layer, the measured value generated by the sampling network is flattened into one dimension through the flat layer, then nodes are expanded to 24567 through the full-connection layer, and then the output of the full-connection layer is adjusted to a specified image size through a reshape operation; the first hidden layer is Transformer Block, wherein Transformer Block comprises the original image and its corresponding position encoding, a PixelNorm normalization layer, a multi-head self-attention mechanism, a PixelNorm normalization layer, and an MLP layer; the second layer of hidden layer is a sub-pixel convolution block, and the sub-pixel convolution block comprises a convolution layer with the size of 3 multiplied by 3, a batch normalization layer, a SELU activation function layer, a sub-pixel convolution layer and a SELU activation function layer.

6. The method of claim 3, wherein the authentication network determines whether the image generated by the generating network is true, and comprises a plurality of convolution layers, a batch normalization layer, a layer and a full connection layer.

7. The method of claim 1, wherein setting hyper-parameters of a transducer generated countermeasure network model comprises: the initial learning rate is set to 0.001, and the number of network iterations is set to 20.

8. A method for reconstructing compressed sensing of an image based on a transform generated against a network according to claim 3, wherein parameters of the generated network are trained and updated by Adam optimization algorithm, and parameters of the identified network are trained and updated by RMSProp optimization algorithm.

9. The method for reconstructing image compressed sensing based on a transform generated countermeasure network according to claim 1, wherein the performance of the network is verified by using an evaluation index peak signal to noise ratio PSNR, structural similarity SSIM.

10. An image compressed sensing reconstruction system based on a Transformer generated countermeasure network, comprising:

the image sample set acquisition module is used for acquiring an image sample set, dividing the image sample set into a training set and a testing set according to a proportion, and preprocessing an image;

the network model construction module is used for constructing a transducer to generate an countermeasure network model according to the sampling rate;

the super-parameter setting module is used for setting a trans-former to generate super-parameters of the countermeasure network model and selecting a loss function and an optimization method;

the training module is used for training the network model under different sampling rates by using the image data set, training the optimal parameters of the learning network model through a loss function and an optimization method, and obtaining the trained Transformer under different sampling rates to generate an countermeasure network model;

and the image reconstruction module is used for performing image compressed sensing reconstruction on the countermeasure network model by using the trained transducer generation and verifying the performance of the network by using the evaluation index.

Images