CN116797461A - Binocular image super-resolution reconstruction method based on multi-level enhanced attention mechanism - Google Patents

Abstract

The application provides a binocular image super-resolution reconstruction method based on a multistage enhanced attention mechanism, which is characterized in that multiple attention is used for feature enhancement in neural network training, feature information in views is fully utilized for fusion processing, a frequency domain related loss function is used for carrying out constraint processing on the frequency domain, the retention of low-frequency information and an image integral structure is enhanced, a better effect is recovered to the binocular image after super-resolution, and clearer textures and edge details are recovered.

Description

Binocular image super-resolution reconstruction method based on multi-level enhanced attention mechanism

Technical field

The present invention relates to the technical field of binocular image super-resolution, in particular to a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism.

Background technique

A very straightforward way to achieve binocular image super-resolution is to use a single-image super-resolution algorithm for the left and right images respectively. Attention mechanism is a very important research direction in the field of deep learning. In the past few years, some single-image super-resolution algorithms with superior performance have emerged, such as RCAN based on channel attention, and pixel attention mechanism based on PAN built, SwinIR built based on the Transformer self-attention mechanism, MAN based on multi-scale large-core attention, etc. However, using only single-image super-resolution methods to independently reconstruct binocular images only uses intra-image self-similarity to recover details, but ignores the additional information that can be exploited across views, that is, cross-view similarity, limitations further improve the super-resolution performance. Therefore, fully utilizing cross-view information can help reconstruct higher-quality super-resolution images, since one view may have complementary information about the same scene area relative to another view. With the needs of society, various super-resolution reconstruction technologies adapted to the times have been proposed. Binocular image super-resolution reconstruction technology has also been applied to the research foundation in various fields, which has great application research significance. This resulted in PASSRnet based on disparity attention, iPASSR based on bidirectional disparity attention mechanism, SwinFSR based on Transformer self-attention mechanism, CVHSSR based on large kernel convolution attention mechanism, etc.

Although there are many attempts to explore combining multiple attention mechanisms with binocular image super-resolution in order to effectively extract more features from internal views and across views, most of the binocular image super-resolution methods are ineffective in recovering images. Natural texture and edge detail are still an open question. Therefore, how to effectively utilize the inter-viewpoint dependence between different attention features based on multiple attention mechanisms to reconstruct super-resolution binocular images requires further exploration.

Contents of the invention

In view of this, the purpose of the present invention is to provide a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism, making full use of the feature information in the view for fusion processing, and using a frequency domain-related loss function to Processing in the domain enhances the retention of low-frequency information and the overall structure of the image, and has a better effect on restoring the binocular image after super-resolution, and restores clearer texture and edge details.

To achieve the above objectives, the present invention adopts the following technical solution: a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism, which includes the following steps:

Step S1, establish a binocular image training set; divide the binocular image super-resolution data set into a training set and a test set. The low-resolution images are generated by bicubic downsampling; in the training phase, the generated low-resolution images are cropped into small patches, the corresponding high-resolution images are also cropped, and these patches are randomly flipped horizontally and vertically to enhance the training data;

Step S2, establish and train a binocular super-resolution reconstruction network model based on a multi-level enhanced attention mechanism; the network uses a pair of low-resolution RGB binocular images as input to generate a super-resolution binocular image;

Step S3, construct a loss function; use the L ₁ loss function and the method combined with the frequency domain loss function to enhance the supervision of high-level feature space and constrain the training of the network;

Step S4, set training parameters for network training;

Step S5, test network performance; use low-resolution binocular image pairs as test samples, input them into the network model trained in the previous step, obtain super-resolution binocular image pairs, and use objective evaluation indicators and visual effect comparison to test the super-resolution resolution effect.

In a preferred embodiment, specifically, the binocular super-resolution reconstruction network model based on the multi-level enhanced attention mechanism includes two left and right weight sharing network branches; in each weight sharing network, the mixed attention information is The extraction modules are stacked to extract the intra-view channels and spatial features of the left and right images; the binocular interactive view attention module is used to capture the global corresponding information and cross-view information extracted from the left and right binocular images; it is divided into three parts: View Intra-feature extraction, interactive view feature fusion and binocular image reconstruction.

In a preferred embodiment, step 2 specifically includes the following steps:

Step S21, in-view feature extraction; in the feature extraction stage, first the input binocular image Input to 3×3 convolution layer to extract shallow features and generate high-dimensional features/> where C is the number of feature channels; then the high-dimensional features are input into the stacked multi-attention enhancement block for intra-view feature extraction to obtain more local features and interactive information and restore more accurate texture details; among them, multi-attention enhancement The block contains a hybrid attention information extraction module and a binocular interactive view attention module;

The hybrid attention information extraction module is the basic module of the left and right branches of the network, which extracts features in the view more deeply by capturing remote and local dependencies; the hybrid attention information extraction module consists of two sequentially connected modules; the first one is a simplified channel and spatial information extraction module, the second one is the feed-forward network module for residual information aggregation; the calculation process of the two parts is as follows:

In the first module, after layer normalization, a 1×1 convolutional layer is used to expand the channels of the input feature map. After that, the generated output will be passed through 3×3 depth convolution to capture the local context of each channel; the cross-activation structure A unit is then used to further learn an effective representation of the spatial context; the next step is to simplify the channel spatial attention module to fully Using the channel attention mechanism and spatial attention mechanism, given the original input/> First, average pooling and 1×1 convolution operations are used to learn the inter-channel relationship of the feature map of a given input image, realize the functions of global spatial information aggregation and channel information interaction, and output simplified channel attention features X ₁ ; through average pooling and maximum pooling operation to aggregate the spatial information of the feature map, and then use a method combining 3×3 convolution and Sigmoid function to obtain a simplified spatial attention map; finally, the input feature map and the output of the Sigmoid layer are multiplied element by element The result of is taken as the output of the simplified spatial attention module X ₂ ; the simplified channel spatial attention module is expressed as:

In the formula, W _C (·), H _AP (·) are respectively 1×1 convolution and average pooling operations, H _AP,1 (·), H _MP,1 (·) respectively represent the averaging in the first dimension. Pooling and maximum pooling operations, H _cat (·) represents splicing in the form of 1 dimension, σ (·) represents Sigmoid activation function respectively, Θ represents element-wise multiplication;

After simplifying the channel and spatial information extraction modules, a 1×1 convolutional inverse transformation is performed on the feature map channels to produce adaptive feature refinement, and the results of the first module are obtained; in the second module, after After the output of the previous module is normalized, a residual information aggregation feed-forward network containing a cross-activation structure B unit is used to improve local context awareness; specifically, given an input tensor First use a 1×1 convolutional layer to extend X′ to a higher dimension/> where k is the expansion ratio; next, a 3×3 depth convolution layer is used to encode the information of X′ ₁ adjacent pixel position, and then the CAS-B unit is used as the activation function of the depth convolution layer, the number of feature channels Halve the output; finally, remap the initial input dimension X′ ₂ through a 1×1 convolution layer;

The above process is expressed as:

In the formula, W _C (·) and W _D are layer normalization, 1×1 convolution, and 3×3 depth convolution respectively, and CAS.B(·) represents the B unit of the cross-activation structure;

Finally, like the previous module, the input of the latter module and the output of the convolutional layer are added as the final result;

Step S22, cross-view feature fusion, uses the binocular interactive view attention module after the mixed attention information extraction module of the left and right branches; the binocular interactive view attention module uses the binocular features generated in the previous step as input to perform two-way cross-view interaction, and Generate interaction features that are fused with the input features of that view; specifically, given the input binocular view features After layer normalization and 1×1 convolution operation, binocular features are obtained and/> Among them/> is a 1×1 convolution; the channel weights are then generated by performing a fast 1D convolution of size k/> Among them, k is determined adaptively through the mapping of the channel dimension C, and the channel weight and the binocular feature are multiplied element by element to obtain the aggregate feature/> As follows:

In the formula, H _MP (·) represents k×k convolution layer and maximum pooling operation respectively, and Θ represents element-wise multiplication;

By calculating the attention matrix once, F _R→L and F _L→R are generated at the same time; finally, the interactive cross-view information and the internal view information FL and FR are fused together through element-by-element addition. According to Formula in/> Is the query matrix of the feature projection in the source view (such as the left view), /> Is the key value matrix of the feature projection within the target view (such as the right view); expressed as follows:

where γ _L and γ _R are trainable channel scaling and initialized to zero to stabilize training,

It is a 1×1 convolution;

Step S23, binocular image reconstruction; after fusion feature extraction, output to 3×3 convolution layer and spatial attention enhancement module, and finally use pixel reorganization operation to upsample the output features to a high-resolution size, and use the global residual path to Use the input binocular image information to further improve the super-resolution performance and restore the super-resolution images of the left and right views.

In the formula, H _C (·), H _E (·), H _P (·), H _↑ (·) are the upsampling operations of convolution operation, enhanced spatial attention module, pixel reorganization and bilinear interpolation respectively;

The enhanced spatial attention module converts the given input Sent to a 1×1 convolutional layer to get where W _C (·) is a 1×1 convolution to reduce the channel size of the input feature; then, the block uses cross-row convolution and cross-row max pooling layers to reduce the spatial size; after a set of convolutions, in order to extract Features are upsampled based on bilinear interpolation to restore the spatial size; combined with residual connectivity, the features are further processed to obtain a 1×1 convolution layer to restore the channel size; finally, the attention matrix is generated by the Sigmoid function and multiplied Take the original input feature X″.

In a preferred embodiment, in step S3, the total difference loss is written as:

L＝L _SR +λL _FFT ,

(10)

In the formula, L _SR and L _FFT respectively represent the frequency domain loss of L ₁ reconstruction loss function and frequency Charbonnier loss, and λ represents the hyperparameter used to control the frequency Charbonnier loss function; the parameter λ of all experiments is set to 0.1;

SR reconstruction loss; SR reconstruction loss is essentially an L ₁ loss function; using the pixel-level L ₁ distance between the super-resolution and ground-truth binocular images, PSNR is obtained; expressed as follows:

In the formula, The super-resolution left and right images generated by the model, /> their high-resolution images respectively;

Frequency domain loss; frequency domain loss that introduces frequency Charbonnier loss; expressed as follows:

In the formula, the constant ε is experimentally set to 10 ^-3 ; FFT(·) represents the fast Fourier transform.

In a preferred embodiment, in step S4, AdamW is used for optimization, where β1=0.9, β2=0.9, and the weight defaults to 0; the learning rate is initially set to 1×10 ^-3 and is reduced through the cosine annealing strategy is 1×10 ^-7 ; the model is trained on 30×90 patches; during the training process, each batch of 32 samples is evenly distributed into 8 parts and iterated 2×10 ⁵ times.

Compared with the existing technology, the present invention has the following beneficial effects: The present invention provides a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism. The present invention constructs a network based on a multi-level enhanced attention mechanism. The model integrates multiple attention mechanisms, comprehensively and efficiently enhances the interaction between in-view information and cross-views, better extracts super-resolution information that cannot be fully utilized in the left and right views of binocular images, and expands the receptive field. while reducing the amount of calculation; using a new cross-attention module and utilizing an efficient channel attention mechanism, a good balance is achieved in efficient interaction; the channel features and spatial feature fusion used are important for forward propagation through the long-range dependencies between features Information, effectively improves the robustness and generalization ability of detection, improves the effect of super-resolution on certain edges, better restores the natural texture of the image, and obtains better super-resolution results with less calculation.

Description of the drawings

Figure 1 is a flow chart of a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism according to a preferred embodiment of the present invention;

Figure 2 is a schematic structural diagram of a binocular super-resolution image network according to a preferred embodiment of the present invention;

Figure 3 is a schematic structural diagram of the hybrid attention information extraction module according to the preferred embodiment of the present invention;

Figure 4 is a schematic diagram of a simplified channel space attention module according to a preferred embodiment of the present invention;

Figure 5 is a schematic diagram of the cross-activation structure of a preferred embodiment of the present invention;

Figure 6 is a schematic diagram of the binocular interactive view attention module according to the preferred embodiment of the present invention;

Figure 7 is a binocular image super-resolution result diagram shown in the preferred embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and examples.

It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meanings commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the exemplary embodiments according to the application; as used herein, the singular form is also intended to include unless the context clearly indicates otherwise. Plural forms, in addition, it should also be understood that when the terms "comprises" and/or "includes" are used in this specification, they indicate the presence of features, steps, operations, means, components and/or combinations thereof.

The present invention provides a binocular image super-resolution method based on a multi-level enhanced attention mechanism, which includes the following four steps:

First, establish a binocular image training set. Using existing public binocular image super-resolution datasets as training samples, low-resolution images are generated by bicubic downsampling. During the training phase, the generated low-resolution images are cropped into small patches, the corresponding high-resolution images are also cropped, and these patches are randomly flipped horizontally and vertically to enhance the training data.

Second, design the network structure. The overall network consists of three parts: intra-view feature extraction, interactive view feature fusion and binocular image reconstruction. In the in-view feature extraction stage, the input binocular images are first input into the convolution layer to extract shallow features and generate high-dimensional features. The high-dimensional features are then input into stacked multi-attention enhancement blocks for intra-view feature extraction to obtain more local features and interactive information and restore more accurate texture details. In the cross-view feature fusion stage, in order to capture the cross information between the left and right views, a binocular interactive view attention module is used after the hybrid attention information extraction module of the left and right branches. The binocular interaction view attention module uses the binocular features generated by the hybrid attention information extraction module in the previous step as input to perform two-way cross-view interaction and generate interaction features that are fused with the input features of the view. In the binocular image reconstruction stage, after fusion feature extraction, the output is output to the convolution layer and spatial attention enhancement module. Finally, the output feature tensor is upsampled using a pixel reorganization operation to restore the super-resolution left and right view images.

Third, construct a loss function. In order to enhance the texture details of binocular images and maintain the disparity consistency between viewpoints, the present invention uses the mean square error loss function and a method combined with the frequency domain loss function to enhance the supervision of high-level feature spaces and constrain the training of the network. The total difference loss can be written as:

L＝L _SR +λL _FFT ,

In the formula, L _SR and L _FFT respectively represent the mean square error loss function and frequency Charbonnier loss function of L ₁ , and λ represents the hyperparameter used to control the frequency Charbonnier loss function. Based on past experience, it is set to 0.1.

Fourth, set training parameters for network training. Select an appropriate optimizer, set parameters such as the loss function, learning rate, maximum number of iterations, and each batch of training sample size, and train the network until the training is completed to obtain the final network weight model;

Fifth, test network performance. Use the low-resolution binocular image pair as a test sample and input it into the network model trained in the previous step to obtain a super-resolution binocular image pair. Use objective evaluation indicators and visual effect comparison to test the super-resolution effect.

Figure 1 is a flow chart of the method of the present invention. For binocular images, follow the detailed steps below to perform super-resolution processing:

Step S1: Establish a binocular image training set. The existing public binocular image super-resolution data set is divided into a training set and a test set, and the low-resolution images are generated by bicubic downsampling. During the training phase, the generated low-resolution images are cropped into small patches, the corresponding high-resolution images are also cropped, and these patches are randomly flipped horizontally and vertically to enhance the training data.

Step S2: Establish and train a binocular super-resolution reconstruction network model based on the multi-level enhanced attention mechanism. As shown in Figure 2, the network takes a pair of low-resolution RGB binocular images as input and generates a super-resolution binocular image. Specifically, the network contains two left and right weight-sharing network branches. In each weight-sharing network, hybrid attention information extraction modules are stacked to extract intra-view channels and spatial features of the left and right images. The binocular interactive view attention module is used to capture the global corresponding information and cross-view information extracted from the left and right binocular images. In general, it can be divided into three parts: intra-view feature extraction, interactive view feature fusion and binocular image reconstruction.

Step S2.1, feature extraction within the view. In the feature extraction stage, the input binocular image is first Input to 3×3 convolution layer to extract shallow features and generate high-dimensional features/> Where C is the number of characteristic channels. The high-dimensional features are then input into stacked multi-attention enhancement blocks for intra-view feature extraction to obtain more local features and interactive information and restore more accurate texture details. Among them, the multi-attention enhancement block includes a hybrid attention information extraction module and a binocular interactive view attention module.

As shown in Figure 3, the hybrid attention information extraction module is the basic module of the left and right branches of the network, which can extract features in the view more deeply by capturing remote and local dependencies. The hybrid attention information extraction module consists of two sequentially connected modules. The first is a simplified channel and spatial information extraction module, and the second is a feed-forward network module for residual information aggregation. The calculation process of the two parts is as follows:

In the first module, after layer normalization, a 1×1 convolutional layer is used to expand the channels of the input feature map. Afterwards, the resulting output is passed through a 3×3 depthwise convolution to capture the local context of each channel. The cross-activation structure unit A (shown in Figure 4) is then used to further learn an effective representation of spatial context. The next step is to simplify the channel spatial attention module, as shown in Figure 5, making full use of the channel attention mechanism and spatial attention mechanism to filter out less useful information, given the original input/> First, average pooling and 1×1 convolution operations are used to learn the inter-channel relationship of the feature map of a given input image, realize the functions of global spatial information aggregation and channel information interaction, and output simplified channel attention features/> The spatial information of the feature map is aggregated through average pooling and maximum pooling operations, and then a simplified spatial attention map is obtained by combining 3×3 convolution and Sigmoid function. Finally, the result of the element-wise multiplication of the input feature map and the output of the Sigmoid layer is used as the output of the simplified spatial attention module. The simplified channel space attention module can be expressed as:

In the formula, W _C (·), H _AP (·) are respectively 1×1 convolution and average pooling operations, H _AP,1 (·), H _MP,1 (·) respectively represent the averaging in the first dimension. Pooling and maximum pooling operations, H _cat (·) means splicing in the form of 1 dimension, σ (·) respectively represents the Sigmoid activation function, and Θ means element-wise multiplication.

After simplifying the channel and spatial information extraction modules, a 1×1 convolutional inverse transformation is performed on the feature mapped channels to produce adaptive feature refinement, and the results of the first module are obtained.

In the second module, after normalizing the output results of the previous module, a residual information aggregation feed-forward network containing a cross-activation structure B unit (shown in Figure 4) is used to improve local context awareness. . Specifically, given an input tensor First extend X′ to higher dimensions using a 1×1 convolutional layer where k is the expansion ratio. Next, a 3×3 depth convolution layer is used to encode the information of the adjacent pixel position of X′ ₁ , and then the CAS-B unit is used as the activation function of the depth convolution layer, and the number of feature channels is halved for output. Finally, it is remapped to the initial input dimension X′ ₂ through a 1 × 1 convolutional layer. The above process can be expressed as:

In the formula, W _C (·) and W _D are layer normalization, 1×1 convolution, and 3×3 depth convolution respectively, and CAS.B(·) represents the B unit of the cross-activation structure.

Finally, like the previous module, the input of the latter module and the output of the convolutional layer are added together as the final result.

Step S2.2, cross-view feature fusion. As shown in Figure 6, in order to capture the cross information between the left and right views, a binocular interactive view attention module is used after the hybrid attention information extraction module of the left and right branches. The binocular interactive view attention module uses the binocular features generated in the previous step as input to perform two-way cross-view interaction and generate interactive features that are fused with the input features of the view. Specifically, given the input binocular view characteristics After layer normalization and 1×1 convolution operation, the binocular features/> and/> Among them/> It is a 1×1 convolution. Then, channel weights are generated by performing a fast 1D convolution of size k/> Among them, k is determined adaptively through the mapping of the channel dimension C, and the channel weight and the binocular feature are multiplied element by element to obtain the aggregate feature/> As follows:

In the formula, Represent k×k convolution layer and maximum pooling operation respectively, and Θ represents element-wise multiplication.

By calculating the attention matrix once, F _R→L and F _L→R are generated simultaneously. Finally, the interactive cross-view information and the internal view information FL and FR are fused together through element-by-element addition, according to Formula in/> Is the query matrix of the feature projection in the source view (such as the left view), /> Is the key value matrix of the feature projection within the target view (such as the right view). The above can be expressed as follows:

where γ _L and γ _R are trainable channel scaling and initialized to zero to stabilize training, It is a 1×1 convolution.

Step S2.3, binocular image reconstruction. After fusion feature extraction, it is output to a 3×3 convolutional layer and spatial attention enhancement module, and finally a pixel reorganization operation is used to upsample the output features to a high-resolution size. In addition, in order to reduce the burden of feature extraction, a global residual path is used in this part to use the input binocular image information to further improve the super-resolution performance and recover the super-resolution images of the left and right views.

In the formula, H _C (·), H _E (·), H _P (·), H _↑ (·) are the upsampling operations of convolution operation, enhanced spatial attention module, pixel reorganization and bilinear interpolation respectively.

The enhanced spatial attention module converts the given input Sent to a 1×1 convolutional layer to get Where W _C (·) is a 1×1 convolution to reduce the channel size of the input feature. Then, the block uses cross-row convolution and cross-row max pooling layers to reduce the spatial size. After a set of convolutions, in order to extract features, upsampling based on bilinear interpolation is performed to restore the spatial size. Combined with residual connectivity, the features are further processed to obtain a 1×1 convolutional layer to restore the channel size. Finally, the attention matrix is generated by the Sigmoid function and multiplied by the original input feature X″.

Step S3, construct the loss function. In order to enhance the texture details of binocular images and maintain the disparity consistency between viewpoints, the present invention uses the L ₁ loss function and a method combined with the frequency domain loss function to enhance the supervision of high-level feature spaces and constrain the training of the network. The total difference loss can be written as:

L＝L _SR +λL _FFT , (10)

In the formula, L _SR and L _FFT respectively represent the frequency domain loss of the L ₁ reconstruction loss function and frequency Charbonnier loss, and λ represents the hyperparameter used to control the frequency Charbonnier loss function. The parameter λ was set to 0.1 for all experiments.

SR reconstruction loss. The SR reconstruction loss is essentially an L ₁ loss function. In order to achieve faster convergence, the present invention uses the pixel-level L ₁ distance between the super-resolution and ground-truth binocular images to avoid overly smooth textures and thereby obtain higher PSNR. It can be expressed as follows:

In the formula, The super-resolution left and right images generated by the model, /> respectively for their high-resolution images.

frequency domain loss. In order to better restore high-frequency details in the image super-resolution task, the present invention introduces the frequency domain loss of frequency Charbonnier loss. It can be expressed as follows:

In the formula, the constant ε is experimentally set to 10 ^-3 . FFT(·) stands for Fast Fourier Transform.

Step S4: Set training parameters for network training. AdamW is used for optimization, where β1=0.9, β2=0.9, and the weight defaults to 0. The learning rate is initially set to 1×10 ^-3 and reduced to 1×10 ^-7 through the cosine annealing strategy. The model is trained on 30×90 patches. During the training process, each batch of 32 samples is evenly distributed into 8 parts, and iterates 2 × 10 ⁵ times.

Step S5, test network performance. Use the low-resolution binocular image pair as a test sample and input it into the network model trained in the previous step to obtain a super-resolution binocular image pair. Use objective evaluation indicators and visual effect comparison to test the super-resolution effect.

In order to show the effect of super-resolution, the experiment compared the bicubic interpolation method (Bicubic), the existing single image super-resolution technology (EDSR) and the binocular image super-resolution technology (PASSRnet, SRResNet+SAM, iPASSR and NAFSSR- L).

The binocular super-resolution image method proposed in the embodiment of the present invention is verified from two perspectives: qualitative and quantitative.

2.1 Qualitative experimental results

In the embodiment of the present invention, a super-resolution operation is performed on the images in the test set, and the results obtained by other methods are compared. As shown in Figure 7, it is the super-resolution result picture. In the image in Figure 7, the box in the lower right corner shows the complete image, and the rest is a partial enlargement of a certain area in the image. It can be seen that compared with other super-resolution methods, the method proposed by the embodiment of the present invention recovers more details related to edges and textures, verifying that the embodiment of the present invention has the ability to perform binocular image super-resolution tasks. good effect.

2.2 Quantitative analysis

The embodiment of the present invention conducted a quantitative error analysis on the super-resolution results of 112 pairs of binocular images in the test set. The compared methods include interpolation method (bicubic), EDSR, (PASSRnet, SRResNet+SAM, iPASSR and NAFSSR-L methods). Objective quality evaluation refers to quantitatively calculating the target image through fixed mathematical formulas, and evaluating the quality of the image based on the calculated value. Currently, the peak signal-to-noise ratio PSNR (Peak Signal-to-Noise Ratio) has a similar structure to SSIM (Structural Similarity Index Measure) is the main objective evaluation index. The PSNR calculation formula of image I and image K is as follows:

Among them, H and W represent the height and width of images I and K respectively, are the pixel values whose position coordinates are x and y in image I, are the pixel peak values of the image, and b are the number of binary digits of the pixel. Currently, in natural image processing Generally, b=8 is taken. The unit of PSNR is decibel (dB), and the value is usually between 20 and 40. The larger the value, the smaller the pixel difference between the reconstructed image and the label image, which proves that the performance of the super-resolution model is better.

Given a labeled image I and a reconstructed image K, the calculation steps of SSIM are as follows:

μ _I and μ _K are the pixel means of I and K respectively, σ _I and σ _K are the variances of I and K respectively, and σ _IK is the covariance of I and K. The value of SSIM is between 0 and 1. The closer the value is to 1, the higher the overall similarity between the two images. SSIM is often used together with PSNR as an objective quality evaluation index.

After testing the images in the test set using different methods and averaging, the experimental results are shown in Table 1:

Table 1. Comparison of PSNR and SSIM of different super-resolution methods on the test set

It can be seen from the results in Table 1 that the binocular image super-resolution method proposed by the embodiment of the present invention achieved an average peak signal-to-noise ratio of 24.21dB and a structural similarity of 0.7633. Compared with other super-resolution methods that use neural networks, this numerical value shows that the super-resolution method proposed in the embodiment of the present invention has better results on the test set, and the mapping relationship between binocular images is used to improve the super-resolution. resolution effect.

Figure 7 shows the reconstruction effect of the present invention compared with the bicubic interpolation method (Bicubic), existing single image super-resolution technology (EDSR) and binocular image super-resolution technology (PASSRnet, SRResNet+SAM, iPASSR and NAFSSR-L) Comparing the super-resolution reconstruction results, the present invention can display the edge lines relatively clearly, and the distinction between the background and the texture body is obvious, and the super-resolution result is good.

Claims (5)

1. The binocular image super-resolution reconstruction method based on the multistage enhanced attention mechanism is characterized by comprising the following steps of:

step S1, a binocular image training set is established; dividing a binocular image super-resolution data set into a training set and a testing set, wherein a low-resolution image is generated through bicubic downsampling; in the training phase, the generated low-resolution images are cut into small blocks, and the corresponding high-resolution images are also cut, and meanwhile, the small blocks are randomly horizontally and vertically flipped to strengthen training data;

step S2, a binocular super-resolution reconstruction network model based on a multi-stage enhanced attention mechanism is established and trained; the network takes a pair of RGB binocular images with low resolution as input to generate a binocular image with super resolution;

s3, constructing a loss function; by L ₁ The loss function and the method combined with the frequency domain loss function enhance the supervision of the high-level characteristic space and restrict the training of the network;

step S4, setting training parameters to perform network training;

s5, testing network performance; and taking the low-resolution binocular image pair as a test sample, inputting the test sample into a network model which is trained in the last step, obtaining the super-resolution binocular image pair, and comparing and checking the super-resolution effect of the binocular image by using objective evaluation indexes and visual effects.

2. The method for reconstructing a binocular image based on a multi-stage enhanced attention mechanism according to claim 1, wherein the binocular super-resolution reconstruction network model based on the multi-stage enhanced attention mechanism comprises two left and right weight sharing network branches; stacking the mixed attention information extraction modules in each weight sharing network, and extracting intra-view channels and spatial features of left and right images; the binocular interaction view attention module is used for capturing globally corresponding information and cross view information extracted from left and right binocular images; the method comprises three parts: intra-view feature extraction, interactive view feature fusion and binocular image reconstruction.

3. The binocular image super-resolution reconstruction method based on the multi-stage enhanced attention mechanism according to claim 2, wherein the step 2 specifically comprises the following steps:

step S21, extracting features in the view; in the feature extraction stage, the input binocular image is first inputInputting into 3×3 convolution layer to extract shallow layer features and generate high-dimensional features +.>Wherein C is the number of characteristic channels; then inputting the high-dimensional features into the stacked multi-attention enhancement blocks for intra-view feature extraction so as to acquire more local features and interaction information and restore more accurate texture details; the multi-attention enhancement block comprises a mixed attention information extraction module and a binocular interactive view attention module;

the mixed attention information extraction module is a basic module of the left and right branches of the network, and the characteristics in the view are extracted more deeply by capturing remote and local dependency relations; the mixed attention information extraction module consists of two modules which are sequentially connected; the first is a simplified channel and spatial information extraction module, and the second is a feedforward network module for residual information aggregation; the two parts of calculation process are as follows:

in the first module, after layer normalization, the channels of the input feature map are extended using a 1×1 convolutional layerThe resulting output will then be passed through a 3 x 3 depth convolution to capture the local context of each channel; the cross-activation structure a unit is then used to further learn the effective representation of the spatial context; the next step is to simplify the channel spatial attention module, make full use of the channel attention mechanism and the spatial attention mechanism, given the original input +.>Firstly, the relation between channels of feature mapping of a given input image is learned by utilizing average pooling and 1 multiplied by 1 convolution operation, the functions of global space information aggregation and channel information interaction are realized, and the simplified channel attention feature X is output ₁ The method comprises the steps of carrying out a first treatment on the surface of the The space information of the feature images is aggregated through the operations of average pooling and maximum pooling, and a method of combining 3×3 convolution and Sigmoid functions is adopted to obtain a simplified space attention graph; finally, the result of element-by-element multiplication of the output of the input feature map and Sigmoid layer is taken as the output X of the simplified spatial attention module ₂ The method comprises the steps of carrying out a first treatment on the surface of the The reduced channel spatial attention module is expressed as:

w in the formula _C (·),H _AP (. Cndot.) are respectively 1X 1 convolution, average pooling operations, H _AP,1 (·),H _MP,1 (. Cndot.) represents the average pooling and maximum pooling operations on dimension 1, H _cat (. Cndot.) represents stitching in the form of dimension 1, σ (. Cndot.) represents Sigmoid activation function, respectively, and Θ represents multiplication by element;

after the simplified channel and the spatial information extraction module, carrying out 1X 1 convolution inverse transformation on the channel mapped by the features to generate self-adaptive feature refinement, and obtaining a result of the first module;

in the second module, after normalizing the output result of the previous module, improving the local context sensing capability through a residual information aggregation feedforward network containing a cross-activation structure B unit; specifically, given an input tensorFirst extending X ' to a higher dimension X ' using a 1X 1 convolution layer ' ₁ Wherein k is the expansion ratio; next, a 3×3 depth convolution layer pair X 'is used' ₁ Coding the information of the adjacent pixel positions, and then using a CAS-B unit as an activation function of the depth convolution layer, and halving the number of characteristic channels and outputting the characteristic channels; finally, the initial input dimension X 'remapped by the 1X 1 convolutional layer' ₂ ；

The above process is expressed as:

w in the formula _C (·),W _D Respectively layer normalization, 1 x 1 convolution, 3 x 3 depth convolution,

CAS.B (-) represents the B unit of the cross-activated structure;

finally, as with the previous module, adding the input of the next module and the output of the convolution layer as the final result;

step S22, cross view feature fusion, wherein a binocular interaction view attention module is used after a left and right branch mixed attention information extraction module; the binocular interaction view attention module uses the binocular characteristics generated in the previous step as input to perform bidirectional cross-view interaction and generates interaction characteristics fused with the view input characteristics; specifically, given an input binocular view characteristicThe binocular feature is obtained by layer normalization and 1X 1 convolution operation>And->Wherein->Is a 1 x 1 convolution; then, a channel weight ++is generated by performing a fast 1D convolution of size k>Wherein k is adaptively determined by mapping of the channel dimension C and the channel weight is multiplied element by the binocular feature to obtain the aggregate feature +.>The following is shown:

in the method, in the process of the application,H _MP (. Cndot.) represents the k x k convolutional layer, max pooling operation, respectively, Θ represents element-by-element multiplication;

by computing the primary attention matrix while generating F _R→L ,F _L→R The method comprises the steps of carrying out a first treatment on the surface of the Finally, the interactive cross-view information is summed and anded by element-wise additionThe intra-view information FL, FR are fused together according toIn->Is a query matrix of feature projections within the source view (e.g. left view), +.>Is a key-value matrix of feature projections within features (e.g., right view) within the target view; the expression is as follows:

/>

wherein, gamma _L And gamma _R Is the trainable channel scaling and initializing to zero to stabilize training,is a 1 x 1 convolution;

step S23, reconstructing a binocular image; after the fusion feature is extracted, the fusion feature is output to a 3X 3 convolution layer and a spatial attention enhancement module, finally, the pixel recombination operation is used for upsampling the output feature to a high resolution size, and a global residual path is used for further improving the super resolution performance by utilizing the input binocular image information, so that the left-right attempted super resolution image is recovered

H in _C (·),H _E (·),H _P (·),H _↑ (. Cndot.) are the upsampling operations of convolution operation, enhanced spatial attention module, pixel reorganization, and bilinear interpolation, respectively;

enhanced spatial attention module will give inputSent to the 1 x 1 convolutional layer to getWherein W is _C (. Cndot.) is a 1 x 1 convolution to reduce the channel size of the input features; the block then uses cross-row rolling and cross-row max pooling layers to reduce the space size; after a set of convolutions, upsampling based on bilinear interpolation to recover the spatial size in order to extract features; combining residual connectivity, and further processing the features to obtain a 1×1 convolutional layer recovery channel size; finally, an attention matrix is generated by the Sigmoid function and multiplied by the original input feature X ".

4. The binocular image super resolution reconstruction method based on the multi-stage enhanced attention mechanism of claim 1, wherein in step S3, the loss of the total difference is written as:

L＝L _SR +λL _FFT ,

wherein L is _SR ,L _FFT Respectively represent L ₁ Reconstructing a frequency domain loss of the loss function, the frequency Charbonnier loss, lambda representing a super parameter for controlling the frequency Charbonnier loss function; the parameter λ for all experiments was set to 0.1;

SR reconstruction loss; the SR reconstruction penalty is essentially an L ₁ Loss ofA function; using pixel level L between super resolution and ground truth binocular image ₁ Distance, PSNR is obtained; the expression is as follows:

in the method, in the process of the application,super-resolution left and right images generated for the model, respectively,>respectively, its high resolution image;

frequency domain loss; introducing frequency domain loss of frequency Charbonnier loss; the expression is as follows:

in the formula, the constant epsilon is experimentally set to be 10 ^-3 The method comprises the steps of carrying out a first treatment on the surface of the FFT (·) represents the fast Fourier transform.

5. The binocular image super resolution reconstruction method based on the multi-stage enhanced attention mechanism of claim 1, wherein in step S4, adamW is adopted for optimization, wherein β1=0.9, β2=0.9, and the weight is default to 0; the learning rate is initially set to 1×10 ^-3 And reduced to 1 x 10 by cosine annealing strategy ^-7 The method comprises the steps of carrying out a first treatment on the surface of the The model was trained on 30 x 90 patches; during training, 32 samples of each batch are evenly distributed to 8 parts, iterating 2×10 ⁵ And twice.