CN112598575B - An image information fusion and super-resolution reconstruction method based on feature processing - Google Patents

Abstract

The invention discloses an image information fusion and super-resolution reconstruction method based on feature processing, and belongs to the technical field of image processing. The invention firstly inputs the high-resolution reference image into an image preprocessing module to obtain a low-resolution reference image. And secondly, inputting the low-resolution input image, the high-resolution reference image and the low-resolution reference image into a feature processing module, and performing feature processing on the input image by the feature processing module to realize feature information matching, transfer and fusion of the high-resolution reference image and the low-resolution input image so as to obtain a fusion feature image. And finally, inputting the low-resolution input image and the fusion characteristic image into a super-resolution reconstruction module to obtain a final super-resolution reconstruction image. The invention adopts the characteristic information of the image instead of the pixel information to carry out information matching and fusion, and can fully utilize the abundant detail texture information carried by the high-resolution reference image, thereby effectively improving the super-resolution reconstruction quality of the low-resolution input image.

Description

An image information fusion and super-resolution reconstruction method based on feature processing

technical field

The invention belongs to the technical field of image processing, and specifically relates to an image information fusion based on feature processing and a super-resolution reconstruction method thereof.

Background technique

Super-resolution reconstruction is a technique that obtains high-resolution images (resolution higher than a specified value) by processing low-resolution images (resolution lower than a specified value). According to the number of input images, it can be divided into single-image-based super-resolution reconstruction algorithms and multi-image-based super-resolution reconstruction algorithms.

The single-image-based image super-resolution reconstruction algorithm directly utilizes the information of the input low-resolution image for reconstruction. This method is simple and easy to implement, but due to the limited information contained in the input low-resolution image itself, the quality of the reconstructed image is also limited.

In order to overcome the above defects, if conditions permit, high-resolution reference images can be introduced into the super-resolution reconstruction algorithm, and the rich texture details contained in the high-resolution images can be fused to the low-resolution images. Improve the quality of super-resolution reconstructions. This is the basic principle of multi-image-based super-resolution reconstruction algorithms.

In order to obtain an ideal super-resolution reconstruction effect, the high-resolution reference image needs to have a high similarity with the low-resolution input image to be processed. At the same time, the information matching and fusion algorithms between the high-resolution reference image and the low-resolution input image have a significant impact on the final super-resolution reconstruction effect. To improve the reconstruction quality of the multi-image-based super-resolution reconstruction algorithm, it is necessary to accurately match the information contained in the high-resolution reference image with the information of the low-resolution input image, and effectively fuse the two information. In this way, the guiding role of high-resolution reference images in super-resolution reconstruction can be fully utilized, thereby improving the quality of super-resolution reconstruction.

However, the existing methods all directly use the pixel information of the image for information matching and fusion. Due to the obvious difference in resolution and sharpness between the high-resolution reference image and the low-resolution input image, coupled with the spatial and temporal differences between the two, the information matching and fusion algorithm based on pixel information has a high probability of mismatching and matching efficiency. Low performance, poor super-resolution reconstruction effect, etc. These problems restrict the performance and stability of multi-image-based super-resolution reconstruction algorithms.

SUMMARY OF THE INVENTION

The purpose of the invention is to propose a feature processing-based image information fusion and its super-resolution method for the deficiencies and defects of the existing multi-image-based super-resolution reconstruction methods, which can effectively improve the low-resolution input image Super-resolution reconstruction quality.

The image information fusion and super-resolution reconstruction method based on feature processing of the present invention includes the following steps:

Perform image preprocessing on the high-resolution reference image and the low-resolution input image (the image to be reconstructed):

down-sampling the high-resolution reference image to obtain a low-resolution reference image that matches the sharpness of the low-resolution input image; and performing the same upsampling on the low-resolution input image and the low-resolution reference image;

Extracting feature information from the high-resolution reference image, the upsampled low-resolution reference image, and the low-resolution input image to obtain a high-resolution reference feature map, a low-resolution reference feature map, and a low-resolution input feature map; And perform image information matching, transfer and fusion processing on the obtained feature map:

Block the high-resolution reference feature map, low-resolution reference feature map, and low-resolution input feature map;

Traverse each sub-block in the low-resolution input feature map, search for the first optimal matching feature sub-block in the low-resolution reference feature map, and based on the space between the high-resolution reference feature map and the low-resolution reference feature map The mapping relationship, based on the image position of the first optimal matching feature sub-block, it is determined that the current sub-block is in the second optimal matching feature sub-block of the high-resolution reference feature map;

Perform feature map reorganization processing based on the image position of each sub-block of the low-resolution input feature map and the second optimal matching feature sub-block to obtain a reorganized feature image;

Image information fusion processing is performed based on the feature map, and the reconstructed feature map is fused with the low-resolution input feature map to obtain a fused feature image:

Perform super-resolution reconstruction on the low-resolution input image based on the fusion feature image to obtain a super-resolution reconstructed image.

Further, the present invention performs the super-resolution reconstruction process processing based on the super-resolution reconstruction network of the coding and decoding structure, wherein the coding part of the super-resolution reconstruction network is used for feature extraction on the low-resolution input image, and splicing through dimensions. The extracted feature image is fused with the fused feature image in the way of method; the decoding part is used to reconstruct the fused feature image and output the super-resolution reconstructed image.

Further, in the present invention, a feature extraction network based on a convolutional neural network is used to perform feature information extraction processing on the high-resolution reference image, the up-sampled low-resolution reference image and the low-resolution input image, and the feature extraction method used is: The network includes several levels of convolutional blocks connected in sequence, and the convolutional blocks are connected by a pooling layer. Each level of convolutional blocks includes several sequentially connected sublayers, each sublayer is composed of sequentially connected convolutional layers and non-convolutional layers. The linear activation layer is composed of a specified layer of nonlinear activation layer in each level of convolution block as the feature map output of the current level, and the multi-level high-resolution reference feature map, low-resolution reference feature map and low-resolution input feature are obtained. Figure; perform image information matching, transfer and fusion processing on the feature maps of the same level to obtain multi-level fusion feature images.

Further, in the super-resolution reconstruction process, the constructed super-resolution reconstruction network includes several encoders and decoders. According to the direction of forward propagation, each encoder is defined as the 1st to Nth level encoders, and the N-level to first-level decoders, where the value of N is the same as the number of convolution block series included in the convolutional neural network for feature information extraction; the first-level encoder and the second-level encoder are connected through a splicing layer , starting from the second-level encoder, each encoder is connected to the down-sampling module and the splicing layer in sequence (that is, the down-sampling module and the splicing layer are sequentially connected between the N-th encoder and the decoder); the input of the splicing layer is also Including the fusion feature image of the specified level (according to the direction of forward propagation, the resolution of the fusion feature image input by each splicing layer decreases in turn), and the feature image extracted by the encoder and the input fusion feature image are fused by dimensional splicing. Processing; the upsampling module and the normalization layer are sequentially connected between the adjacent decoders, and the rth encoder is also connected to the r-1th decoder by skip connection (that is, the output of the rth encoder is connected to the connection. The output of the normalization layer of the r-th decoder is superimposed and then input to the r-1-level decoder), where 1<r≤N; the first-level decoder is connected to the reconstruction layer, which is used for the first-level decoder. The feature image output by the decoder is reconstructed to obtain a super-resolution reconstructed residual image, and finally the super-resolution reconstructed residual image and the low-resolution input image are superimposed to obtain the final super-resolution reconstructed image. That is, in the present invention, the output of each level of encoder is equipped with a splicing layer for splicing and fused feature maps; except for the first-level encoder, each level of encoder is equipped with a downsampling layer for downsampling the feature map image. Sampling, resizing; except the first-level decoder, each level of decoder is configured with an upsampling layer and a normalization layer.

Further, when down-sampling the high-resolution reference image, a pyramid down-sampling method can be used. In the up-sampling process, a simple interpolation algorithm can be used, or an existing single-image-based image super-resolution reconstruction method can be used to perform up-sampling and preliminary super-resolution reconstruction.

Further, when the image information fusion processing is performed based on the feature map, the two feature sub-blocks to be matched are regarded as vectors, that is, the image information of the feature sub-blocks is vectorized, and then the similarity between the two vectors is measured based on the vector similarity. The similarity of the vector, where the vector similarity can be obtained by the weighted summation of the vector cosine distance and the vector Manhattan distance after normalization respectively. The greater the vector similarity, the greater the similarity between the two feature sub-blocks. For a given feature sub-block, when the vector similarity reaches the maximum value, the corresponding candidate feature sub-block is the optimal matching feature sub-block of the given feature sub-block.

Further, when performing image information fusion processing based on the feature map, the normalized low-resolution input feature map and the reorganized feature image (based on the image position of each sub-block of the low-resolution input feature map and the second highest The optimal matching feature sub-blocks are obtained by performing feature map recombination) and linear fusion is performed, and the linear fusion result is standardized to obtain a fusion feature image. That is, the fusion processing of image information is realized based on the overall migration of the data distribution space.

Further, when performing image information fusion processing based on feature maps, it can also be implemented based on linear guided filtering. The low-resolution input feature map is used as input, the reconstructed feature image is used as a guided template, and the output of linear guided filtering is the fusion feature image. . The fused feature image is generally similar to the low-resolution input feature map, but it integrates the key feature information of the bootstrap template (ie, the reconstructed feature image).

To sum up, due to the adoption of the above-mentioned technical solutions, the beneficial effects of the present invention are:

Compared with the existing image fusion and super-resolution methods, the present invention uses the feature information extracted by the convolutional neural network to replace the pixel information to perform image block matching, and realizes information matching, transfer and fusion between images of different resolutions in the feature domain. . This can completely and accurately transfer and fuse the high-resolution reference image information to the low-resolution input image, which significantly improves the super-resolution reconstruction effect of the low-resolution input image.

The invention avoids various drawbacks of traditional pixel-based image information matching and fusion and multi-image-based image super-resolution methods, solves the problem of cross-resolution image matching, and is not limited by the size and number of images.

Description of drawings

1 is a schematic diagram of the processing process of the reconstruction process of the present invention in a specific embodiment;

2 is a schematic diagram of a preprocessing process in a specific embodiment;

3 is a schematic diagram of a feature extraction network structure in a specific embodiment;

4 is a flowchart of feature matching and transfer processing in a specific embodiment;

5 is a flow chart of feature fusion in a specific embodiment;

FIG. 6 is a schematic diagram of an optimized best matching feature search method in a specific embodiment.

FIG. 7 is a schematic diagram of a super-resolution reconstruction process in a specific embodiment, wherein FIG. 7( a ) is a structural diagram of a super-resolution reconstruction network, and FIG. 7( b ) is a schematic structural diagram of an encoder/decoder.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

The image information fusion and super-resolution reconstruction method based on feature processing in the present invention uses the feature information of the image instead of the pixel information to perform information matching and fusion, and can fully utilize the rich detailed texture information carried by the high-resolution reference image. In turn, the quality of super-resolution reconstruction of low-resolution input images is effectively improved. Referring to FIG. 1 , the reconstruction method of the present invention requires one low-resolution input image and at least one high-resolution input image as inputs to the reconstruction method. First, image preprocessing is performed to obtain a low-resolution reference image and a pre-processed low-resolution input image. Secondly, feature extraction is performed on the low-resolution input image, the high-resolution reference image, and the low-resolution reference image to obtain their corresponding feature images. Next, perform feature information matching, transfer and fusion processing on the high-resolution reference image and the low-resolution input image to obtain a fused feature image. Finally, perform super-resolution reconstruction processing on the low-resolution input image and the fusion feature image to obtain the final super-resolution reconstructed image.

Referring to FIG. 2 , in this specific implementation manner, an image preprocessing method based on a high-resolution reference image is specifically: performing Gaussian pyramid downsampling on the high-resolution reference image to obtain a low-resolution reference image. Afterwards, the low-resolution reference image and the low-resolution input image are each upsampled by bicubic spline interpolation to expand the physical size. The downsampling factor and the upsampling factor of the high-resolution reference image are reciprocal of each other, thereby ensuring that the physical size of the low-resolution reference image and the high-resolution reference image remains the same after upsampling. At the same time, the upsampling factor of the low-resolution reference image and the low-resolution input image remains the same, so that the two are similar in pixel sharpness.

Optionally, during image preprocessing, the existing and mature single-image-based super-resolution methods can be used to perform upsampling operations, so that preliminary single-image super-resolution reconstruction can be achieved while upsampling. The upsampling method is customizable. The introduction of the existing single-image-based super-resolution network can ensure that the final super-resolution reconstruction effect is not inferior to the single-image-based super-resolution reconstruction method in the worst case. At the same time, from an optimization point of view, this process helps the super-resolution reconstruction network to converge faster during training.

In this specific embodiment, the feature extraction processing of the image is realized through a neural network. The constructed feature extraction network is essentially a convolutional neural network, which includes several levels of convolution blocks, and the convolution blocks are connected by a pooling layer. Convolution blocks at all levels include a certain number of convolution and nonlinear activation processing, as shown in Figure 3, where conv refers to the convolution layer, relu refers to the ReLU activation layer, and pooling refers to the pooling layer. The pooling multiplier of the pooling layer is 2, and convi_j and relui_j are used to distinguish different convolutional layers and ReLU activation layers, where i represents the level, and j represents the corresponding number of layers. And in the feature extraction process, in order to ensure that the input and output image sizes of each layer are consistent, all convolutional layers need to perform edge 0 processing.

In this specific implementation, the outputs of the three layers of relu1_1, relu2_1 and relu3_1 are selected as the feature images extracted by the feature extraction network, which are respectively named as the first-level feature image relu1_1, the second-level feature image relu2_1 and the third-level feature image relu3_1. These three-level features represent three different levels of image features. Assuming that the size of the input image is H*W*3 (RGB three-channel image), then the physical size of the output three-level feature image is H*W*C ₁ , (H/2)*(W/2)* C ₂ and (H/4)*(W/4)*C ₃ . where C ₁ , C ₂ and C ₃ represent the number of channels of different levels of feature images, respectively.

Optionally, in the process of implementing the feature extraction network shown in FIG. 3 , the present invention supports importing parameters from an external image recognition convolutional neural network as parameters of the feature extraction network. Existing image recognition convolutional neural networks (such as VGG network) generally contain image feature extraction components. After a long period of application, the parameters of these components have become relatively stable and mature, and can be used directly without worrying about the instability of image feature extraction. Compared with re-customizing the feature extraction network, the parameters imported into the mature network can not only effectively reduce the workload, but will not have obvious adverse effects on the final super-resolution reconstruction results.

Referring to FIG. 4 , in the process of feature matching and transfer, in this specific embodiment, the feature map R0_F of the high-resolution reference image, the feature map R1_F of the low-resolution reference image and the feature map of the low-resolution input image need to be combined first. LR_F is split separately to obtain several feature sub-blocks. Each feature sub-block is treated as a vector, has the same physical size and may spatially coincide with each other. Given the i-th feature sub-block LR _i in the feature image LR_F, for each feature sub-block in the feature image R1_F, a feature matching algorithm is defined in the present invention, and the two feature sub-blocks are measured by calculating the vector similarity VS. Similarity between blocks. Among them, the calculation formula of VS is:

这样可以消除R1_j的模带来的影响，因而向量内积IP的计算结果才能在数学上等价为向量的余弦距离。同时为了克服IP和MD两者在度量上的不一致，在计算IP和MD的数值后需要各自进行标准化操作。具体来讲，对于给定的LR_i，计算特征图像R1_F中所有的候选特征子块R1_j的IP值和MD值。对所有候选特征子块的IP值和MD值分别进行标准化，标准化后度量上的差异即可消除。随后再进行加权和得到最后的VS值。Among them, LR _i represents the i-th feature sub-block in the feature image LR_F, R1 _j represents the j-th feature sub-block in the feature image R1_F, and w represents the weighted weight between 0 and 1. IP represents the inner product of the vector, and MD represents the Manhattan distance (aka city distance) of the vector. ‖*‖ represents the modulo length of the vector, and ‖*‖ _norm represents the normalization process. When calculating IP, R1 _j needs to be converted into a unit modulo length vector

In this way, the influence of the modulus of R1 _j can be eliminated, so the calculation result of the vector inner product IP can be mathematically equivalent to the cosine distance of the vector. At the same time, in order to overcome the inconsistency in the measurement of IP and MD, it is necessary to perform normalization operations respectively after calculating the values of IP and MD. Specifically, for a given LR _i , the IP and MD values of all candidate feature sub-blocks R1 _j in the feature image R1_F are calculated. The IP values and MD values of all candidate feature sub-blocks are normalized respectively, and the differences in metrics can be eliminated after normalization. The weighted sum is then performed to obtain the final VS value.

代表R1_F中最佳匹配特征子块。当VS_(i,j)取到最大时，对应的j即为j^*。数学表达式为：It can be seen from the VS formula that the larger the VS value, the higher the similarity of the two feature sub-blocks. Denote j ^* as the feature sub-block label with the highest similarity with the i-th feature sub-block LR _i in the given feature image LR_F in the feature image R1_F,

Represents the best matching feature sub-block in R1_F. When VS _(i,j) takes the maximum, the corresponding j is j ^* . The mathematical expression is:

便已经确定，这里

称为第一最优匹配特征子块。之后利用R0_F与R1_F之间的空间位置映射关系，即可得到R0_F中第j^*个特征子块

是LR_F中第i个特征子块LR_i的最佳匹配特征子块，这里

称为第二最优匹配特征子块。从信息匹配的角度讲，第二最优匹配特征子块

所包含的信息正是LR_F中第i个特征子块LR_i所需要的。After solving for j ^* , the j ^* th feature sub-block on R1_F

It has been determined that here

It is called the first optimal matching feature sub-block. Then, using the spatial position mapping relationship between R0_F and R1_F, the j ^* th feature sub-block in R0_F can be obtained

is the best matching feature sub-block of the i-th feature sub-block LR _i in LR_F, here

It is called the second optimal matching feature sub-block. From the point of view of information matching, the second optimal matching feature sub-block

The information contained is exactly what is needed by the i-th feature sub-block LR _i in LR_F.

将会填充到重组特征图像RC_F特定的位置中。重组特征图像的尺寸与LR_F的尺寸相同，

在RC_F中的填充位置为LR_F中第i个特征子块LR_i在LR_F中的相对位置。The last step is feature transfer, the second best matching feature sub-block

will be filled into a specific position of the reconstructed feature image RC_F. The size of the reconstructed feature image is the same as that of LR_F,

The filling position in RC_F is the relative position of the i-th feature sub-block LR _i in LR_F in LR_F.

For each feature sub-block in LR_F, the second best matching feature sub-block in R0_F can always be found through the above matching method. Then, through the relative position of the given feature sub-block, the second optimal matching feature sub-block is filled to the corresponding position of the reconstructed feature image. Since different feature sub-blocks may overlap each other during the process of splitting feature sub-blocks. Therefore, when the second optimal matching feature sub-blocks are filled into the reconstructed feature image, there will also be overlapping situations. Further, when multiple feature sub-blocks overlap each other in space, the conflict can be resolved by calculating the average value of the overlapping elements in the spatial position.

Referring to Figure 5, in the process of feature fusion, four variables need to be calculated first: the mean and variance of the reconstructed feature image RC_F, and the mean and variance of the low-resolution input feature map LR_F. After obtaining these four variables, the fusion feature image M_F is calculated by the feature fusion formula:

M_F=||μ*||LR_F|| _norm +(1-μ)*||RC_F|| _norm || _norm

Among them, ||*|| _norm is a standardization operation, and the mean and variance of the input data need to be calculated during standardization. μ is a weighted weight between 0 and 1.

Optionally, in the process of feature fusion, a feature fusion method based on guided filtering can also be used to implement, and the specific implementation process is as follows:

Let the low-resolution input feature map be p and the reconstituted feature map be I, given the window radius r and the regularization parameter ε. Among them, the window radius r indicates that the area when the mean is calculated is an area of r*r.

Calculate the mean value image mean _p =f _mean (p) of the feature map p, and the mean value image mean _I =f _mean (I) of the reorganized feature map I;

Take the square of the pixel value of each pixel of the reorganized feature map I to obtain the image I ² , and calculate its mean image corr _I =f _mean (I.*I), and set the reorganized feature map as I and the feature map p in the same position Multiply the pixel values of , to obtain the image I.*p and calculate its mean image corrI _p =f _mean (I.*p); where A.*B represents multiplying the pixel values of the same pixel position in images A and B .

Calculate the variance image var _I =corr _I -mean _I .*mean _I of the reconstructed feature map I, and the covariance image cov _Ip =corr _Ip -mean _I .*mean _{p of the reconstructed feature map I and the feature map p} ;

Calculate the parameter a according to the formula a=cov _Ip /(var _I +ε), and calculate the parameter b according to the formula b=mean _p -a.*mean _I , so as to obtain the respective mean values mean _a and mean _b of the parameters a and b, and finally The fusion feature map q is obtained according to the formula q=mean _a .*I+mean _b .

Optionally, in order to speed up the matching process, the feature matching process in the present invention only matches the third-level feature image relu31. For the first-level feature image and the second-level feature image, matching can be performed according to the matching result of the third-level feature image and the spatial correspondence. When two feature sub-blocks in the third-level feature image are successfully matched, the matching result can be mapped to the first-level feature image and the second-level feature image.

Optionally, in the process of calculating the best matching feature sub-block, on the premise that the similarity between the low-resolution input image and the high-resolution reference image is sufficiently high (greater than or equal to the specified similarity threshold), the adjacent The matching position information of the feature sub-blocks is used to constrain the search range, which helps to greatly reduce the amount of computation in the matching process and speed up the operation. Referring to FIG. 6 , the optimized best matching feature search process of the present invention is specifically: for a certain feature sub-block X on LR_F, the first best matching feature sub-block on R1_F is XO. For the adjacent feature sub-block Y of X, the center position of the last first best matching feature sub-block XO can be relatively offset (the specific offset is set according to specific processing requirements), and then the search anchor point can be obtained. Then, a relatively small area is determined by the position of the anchor point to search for the first best matching sub-block Y0 of Y. The search area is usually set as a rectangle, and the specific range of the search area is determined based on the distance from the configured anchor point to each side of the rectangular search area. Experiments show that in most cases the best matching results obtained by this fast search method are consistent with the results obtained by the complete full-graph search method described in Figure 4, and from the perspective of final reconstruction, 2 There is no significant difference in the final reconstructed renderings of each matching scheme.

In this specific embodiment, the super-resolution reconstruction process is implemented by a convolutional network. Referring to FIG. 7( a ), the super-resolution reconstruction convolutional neural network adopted is an encoding-decoding structure as a whole. The first half of the network has 3 encoders, where the second-level encoder and the third-level encoder are followed by a downsampling module and a Batch Normalization normalization layer. The downsampling module is a convolutional layer with a stride of 2, which can adjust the length and width of the feature image output by the encoder to 1/2 of the original, and keep the physical size of the fused feature image spliced in from the outside. In the stitching layer, the feature image of the original network and the fused feature image are dimensionally stitched in the last dimension. For example, assuming the original network feature image size is B*H*W*C ₁ , and the fusion feature image size is B*H*W*C ₂ , then the size of the stitched feature image is B*H*W*(C ₁ +C ₂ ), where B, H, and C represent the number of images in each batch, the height of the image, and the width of the image, respectively, and C ₁ and C ₂ represent the number of feature image channels. The second half of the network corresponds to three decoders, of which the second-level decoder and the third-level decoder are followed by an upsampling module and a Batch Normalization normalization layer. The upsampling module is a deconvolutional layer with a stride of 2, which is used to adjust the length and width of the output feature image of the decoder to 2 times the original. There are also three long-distance jump connection layers in the network. The last layer of the network is the reconstruction layer. The super-resolution reconstruction residual image is obtained by reconstructing the input feature image. The reconstruction layer can be set to a step size of 1. For the convolutional layer, the number of output channels is adjusted based on the actual situation. For example, for RGB images, the number of output channels is set to 3; for a single Y-channel image (called luminance image), the number of output channels is 1; finally, the super-resolution reconstruction is performed The residual image is superimposed with the original input image to obtain the final super-resolution reconstruction result. In the process of super-resolution reconstruction, in order to ensure that the input and output image sizes of each layer are consistent, all convolutional layers except the downsampling module need to perform edge-filling 0 processing.

Referring to Figure 7(b), the internal structure of the encoder/decoder is: each encoder and decoder consists of 5 convolutional layers conv and 1 Batch Normalization normalization layer. The activation function of each convolutional layer is the pReLU function. The convolutional layers are densely connected to each other, and there is a skip connection layer, which adds the output of the first convolutional layer and the output of the last convolutional layer to obtain the final output. Dense connections can reuse image features and improve the performance of the network. The skip connection allows the network to learn the residuals of the input and expected output rather than the expected output itself during the training process, which can effectively reduce the network training burden and also has a positive effect on network performance improvement.

In this specific embodiment, the training and application (ie, super-resolution reconstruction) of the super-resolution reconstruction network are respectively:

In the training process, the training data is first prepared and divided into training set and test set. The training set is used to train and update the parameters of the super-resolution reconstruction network, and the test set is used to evaluate the network performance. In this specific implementation manner, the feature extraction network imports some parameters of the external network, so it is not updated. In order to speed up the training, the offline feature processing process is performed first, and the data of the fusion feature image is saved as the intermediate data. This data is then imported when training the super-resolution reconstruction network to update the network parameters and save the network model. The parameters of the saved super-resolution reconstruction network can be exported after training.

In the application stage, for a given high-resolution reference image and a low-resolution input image, image preprocessing and feature processing are performed to obtain fused feature image data, which are then sent to the super-resolution reconstruction network for super-resolution. rate reconstruction to obtain the final result.

The above descriptions are only specific embodiments of the present invention, and any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all the disclosed features, or All steps in a method or process, except mutually exclusive features and/or steps, may be combined in any way.

Claims (10)

1. an image information fusion and super-resolution reconstruction method based on feature processing, is characterized in that, comprises the following steps: Perform image preprocessing on the high-resolution reference image and the low-resolution input image: downsample the high-resolution reference image to obtain a low-resolution reference image that matches the clarity of the low-resolution input image; The same upsampling is performed on the low-resolution input image and the low-resolution reference image; Extracting feature information from the high-resolution reference image, the upsampled low-resolution reference image, and the low-resolution input image to obtain a high-resolution reference feature map, a low-resolution reference feature map, and a low-resolution input feature map; And perform image information matching, transfer and fusion processing on the obtained feature map: Perform block processing on high-resolution reference feature maps, low-resolution reference feature maps, and low-resolution input feature maps; Traverse each feature sub-block in the low-resolution input feature map, search for the first optimal matching feature sub-block in the low-resolution reference feature map, and based on the difference between the high-resolution reference feature map and the low-resolution reference feature map The spatial mapping relationship, based on the image position of the first optimal matching feature sub-block, determine the second optimal matching feature sub-block of the current sub-block in the high-resolution reference feature map; Perform feature map reorganization processing based on the image position of each sub-block of the low-resolution input feature map and the second optimal matching feature sub-block to obtain a reorganized feature image; Perform image information fusion processing based on the recombination feature map to obtain a fusion feature image; Perform super-resolution reconstruction on the low-resolution input image based on the fusion feature image to obtain a super-resolution reconstructed image. 2. The method according to claim 1, wherein the super-resolution reconstruction process is processed by a super-resolution reconstruction network based on an encoding and decoding structure, wherein the encoding part of the super-resolution reconstruction network is used for low-resolution reconstruction. Feature extraction is performed on the input image, and the extracted feature image is fused with the fused feature image by dimension stitching; the decoding part is used to reconstruct the fused feature image and output the super-resolution reconstructed image. 3. The method of claim 1, wherein a feature extraction network based on a convolutional neural network is used to perform feature information on a high-resolution reference image, an upsampled low-resolution reference image and a low-resolution input image. For the extraction process, the convolutional neural network used includes several levels of convolutional blocks connected in sequence, and the convolutional blocks are connected by a pooling layer. It consists of sequentially connected convolutional layers and non-linear activation layers. A layer of non-linear activation layers is designated from each level of convolution block as the feature map output of the current level, and multi-level high-resolution reference feature maps and low-resolution reference feature maps are obtained. Feature map and low-resolution input feature map; and perform image information matching, transfer and fusion processing on the feature maps of the same level to obtain multi-level fusion feature images. 4. The method according to claim 2, wherein the super-resolution reconstruction network comprises several encoders and decoders, and according to the direction of forward propagation, each encoder is sequentially defined as the first level to the Nth level Encoder, N-th to 1st-level decoder, where the value of N is the same as the number of convolution block series included in the convolutional neural network for feature information extraction processing; between the first-level encoder and the second-level encoder Through the splicing layer connection, starting from the second-level encoder, each encoder is connected to the downsampling module and the splicing layer in sequence; the input of the splicing layer also includes the fusion feature image of the specified level. The feature image and the input fused feature image are fused; the adjacent decoders are connected to the upsampling module and the normalization layer in sequence, and the rth encoder is also connected to the r-1th decoder by skip connection. Where 1<r≤N; the first-level decoder is connected to the reconstruction layer, which is used to reconstruct the feature image output by the first-level decoder to obtain a super-resolution reconstruction residual image, and finally the super-resolution reconstruction residual image is obtained. The difference image is superimposed with the low-resolution input image to obtain the final super-resolution reconstructed image. 5. The method according to claim 3, wherein, for the multi-level high-resolution reference feature map, the low-resolution reference feature map and the low-resolution input feature map, feature sub-block matching is performed on the feature maps of the same level. During processing, only the feature block matching processing of the first and second optimal matching feature sub-blocks is performed on the feature map of the last level of the convolutional neural network in the feature information extraction process, and the feature block matching between the feature maps of the remaining levels is performed. processing, and the corresponding matching result is obtained directly based on the spatial mapping relationship of the feature block matching processing result between the last-level feature maps. 6. The method according to claim 1, wherein when performing image information fusion processing based on the feature map, the two feature sub-blocks to be matched are used as vectors, and the vector similarity is used as the matching of the two feature sub-blocks The first optimal matching feature sub-block of the feature sub-blocks in the low-resolution input feature map is searched based on the matching degree between the feature sub-blocks. 7 . The method according to claim 6 , wherein the vector similarity is obtained by performing a weighted summation on the vector cosine distance and the vector Manhattan distance of the two vectors after normalization respectively. 8 . 8. The method according to claim 1, wherein, when performing image information fusion processing based on the feature map, the image position of each sub-block of the low-resolution input feature map and the second optimal matching feature sub-block are performed. The feature map is reconstructed to obtain a reconstructed feature image, and then the low-resolution input feature map and the reconstructed feature image that have been standardized respectively are linearly fused, and the linear fusion result is standardized to obtain a fused feature image. 9. The method according to claim 1, wherein, when performing image information fusion processing based on feature maps, it is realized by means of linear guided filtering, using low-resolution input feature maps as input, and recombining feature images as guide templates , the output of linear guided filtering is the fusion feature image. 10. The method of claim 1, wherein when searching for the first optimal matching feature sub-block of the feature sub-block in the low-resolution input feature map, if the low-resolution input image and the high-resolution reference When the similarity between images is greater than or equal to the specified similarity threshold, the search range of the first optimal matching feature sub-block is constrained based on the matching position information of adjacent feature sub-blocks: A certain feature sub-block in the low-resolution input feature map is defined as feature sub-block X, and its first optimal matching feature sub-block in the low-resolution reference feature map is X0; For the adjacent feature sub-block Y of the feature sub-block X, the relative offset of the center position of X0 is performed to obtain the search anchor point; Based on the distance from the configured search anchor point to each boundary of the search range of the first optimal matching feature sub-block, the search range of the first optimal matching feature sub-block of the feature sub-block Y is determined.

Images