CN112598575B - Image information fusion and super-resolution reconstruction method based on feature processing - Google Patents
Image information fusion and super-resolution reconstruction method based on feature processing Download PDFInfo
- Publication number
- CN112598575B CN112598575B CN202011528460.3A CN202011528460A CN112598575B CN 112598575 B CN112598575 B CN 112598575B CN 202011528460 A CN202011528460 A CN 202011528460A CN 112598575 B CN112598575 B CN 112598575B
- Authority
- CN
- China
- Prior art keywords
- image
- feature
- resolution
- low
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000004927 fusion Effects 0.000 title claims abstract description 44
- 238000012545 processing Methods 0.000 title claims abstract description 41
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 230000008569 process Effects 0.000 claims description 26
- 239000013598 vector Substances 0.000 claims description 23
- 238000000605 extraction Methods 0.000 claims description 21
- 238000010586 diagram Methods 0.000 claims description 19
- 238000005070 sampling Methods 0.000 claims description 18
- 238000007499 fusion processing Methods 0.000 claims description 13
- 238000010606 normalization Methods 0.000 claims description 13
- 238000013527 convolutional neural network Methods 0.000 claims description 11
- 230000004913 activation Effects 0.000 claims description 8
- 238000011176 pooling Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 4
- 230000008521 reorganization Effects 0.000 claims description 3
- 230000000903 blocking effect Effects 0.000 claims description 2
- 238000012546 transfer Methods 0.000 abstract description 6
- 238000012549 training Methods 0.000 description 12
- 230000000694 effects Effects 0.000 description 6
- 230000007547 defect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 101100134058 Caenorhabditis elegans nth-1 gene Proteins 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image information fusion and super-resolution reconstruction method based on feature processing, and belongs to the technical field of image processing. The invention firstly inputs the high-resolution reference image into an image preprocessing module to obtain a low-resolution reference image. And secondly, inputting the low-resolution input image, the high-resolution reference image and the low-resolution reference image into a feature processing module, and performing feature processing on the input image by the feature processing module to realize feature information matching, transfer and fusion of the high-resolution reference image and the low-resolution input image so as to obtain a fusion feature image. And finally, inputting the low-resolution input image and the fusion characteristic image into a super-resolution reconstruction module to obtain a final super-resolution reconstruction image. The invention adopts the characteristic information of the image instead of the pixel information to carry out information matching and fusion, and can fully utilize the abundant detail texture information carried by the high-resolution reference image, thereby effectively improving the super-resolution reconstruction quality of the low-resolution input image.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a feature processing-based image information fusion and super-resolution reconstruction method thereof.
Background
Super-resolution reconstruction is a technique of obtaining a high-resolution image (resolution higher than a certain specified value) by processing a low-resolution image (resolution lower than a certain specified value). The input images are classified according to the number thereof, and may be classified into a single-map-based super-resolution reconstruction algorithm and a multi-map-based super-resolution reconstruction algorithm.
The single-image-based image super-resolution reconstruction algorithm directly utilizes the information of the input low-resolution image for reconstruction. The method is simple and easy to implement, but the quality of the image reconstructed by the method is limited because the input low-resolution image contains limited information.
In order to overcome the defects, under the condition that conditions allow, the high-resolution reference image can be introduced into a super-resolution reconstruction algorithm, rich texture detail information contained in the high-resolution image is fused onto the low-resolution image, and the quality of super-resolution reconstruction is improved. This is the basic principle of the multi-map based super-resolution reconstruction algorithm.
In order to obtain a relatively ideal super-resolution reconstruction effect, the high-resolution reference image needs to have a higher similarity with the low-resolution input image to be processed. Meanwhile, the information matching and fusion algorithm between the high-resolution reference image and the low-resolution input image has a great influence on the final super-resolution reconstruction effect. To improve the reconstruction quality of the multi-map-based super-resolution reconstruction algorithm, the information contained in the high-resolution reference image needs to be accurately matched with the information of the low-resolution input image, and the information of the high-resolution reference image and the information of the low-resolution input image need to be effectively fused. Therefore, the guidance function of the high-resolution reference image in super-resolution reconstruction can be fully exerted, and the super-resolution reconstruction quality is improved.
However, the existing methods directly use the pixel information of the image for information matching and fusion. Due to the fact that the resolution and the definition of the high-resolution reference image and the low-resolution input image are obviously different and the space-time difference of the high-resolution reference image and the low-resolution input image is added, the problems that the mismatching probability is high, the matching efficiency is low, the super-resolution reconstruction effect is poor and the like exist in the pixel information based information matching and fusion algorithm. These problems limit the performance and stability of the multi-map based super-resolution reconstruction algorithm.
Disclosure of Invention
The invention aims to: aiming at the defects of the existing multi-image-based super-resolution reconstruction method, the image information fusion based on the feature processing and the super-resolution method thereof are provided, and the super-resolution reconstruction quality of the low-resolution input image is effectively improved.
The invention discloses a feature processing-based image information fusion and super-resolution reconstruction method, which comprises the following steps:
image pre-processing of a high resolution reference image and a low resolution input image (image to be reconstructed):
down-sampling the high-resolution reference image to obtain a low-resolution reference image matched with the definition of the low-resolution input image; and performing the same upsampling on the low-resolution input image and the low-resolution reference image;
extracting feature information of the high-resolution reference image, the up-sampled low-resolution reference image and the low-resolution input image to obtain a high-resolution reference feature map, a low-resolution reference feature map and a low-resolution input feature map; and carrying out image information matching, transferring and fusing on the obtained characteristic diagram:
carrying out blocking processing on the high-resolution reference characteristic diagram, the low-resolution reference characteristic diagram and the low-resolution input characteristic diagram;
traversing each sub-block in the low-resolution input feature map, searching a first optimal matching feature sub-block in the low-resolution reference feature map, and determining that the current sub-block is located in a second optimal matching feature sub-block of the high-resolution reference feature map based on the image position of the first optimal matching feature sub-block based on the spatial mapping relation between the high-resolution reference feature map and the low-resolution reference feature map;
performing feature map reorganization processing on the basis of the image position of each sub-block of the low-resolution input feature map and the second optimal matching feature sub-block to obtain a reorganized feature image;
carrying out image information fusion processing based on the feature map, fusing the recombined feature map and the low-resolution input feature map to obtain a fusion feature image:
and performing super-resolution reconstruction on the low-resolution input image based on the fusion characteristic image to obtain a super-resolution reconstruction image.
Further, the super-resolution reconstruction network based on the coding and decoding structure carries out super-resolution reconstruction process processing, wherein a coding part of the super-resolution reconstruction network is used for carrying out feature extraction on a low-resolution input image and fusing the extracted feature image with a fused feature image in a dimension splicing mode; the decoding part is used for reconstructing the fused characteristic image and outputting a super-resolution reconstructed image.
Further, in the invention, a feature extraction network based on a convolutional neural network is adopted to extract feature information of a high-resolution reference image, an up-sampled low-resolution reference image and a low-resolution input image, the adopted feature extraction network comprises a plurality of stages of convolution blocks which are connected in sequence, the convolution blocks are connected through a pooling layer, each stage of convolution block comprises a plurality of sublayers which are connected in sequence, each sublayer consists of a convolution layer and a nonlinear activation layer which are connected in sequence, and a designated nonlinear activation layer in each stage of convolution block is used as a feature map of the current stage to be output to obtain a multi-stage high-resolution reference feature map, a low-resolution reference feature map and a low-resolution input feature map; and carrying out image information matching, transferring and fusion processing on the characteristic diagrams of the same level to obtain a multi-level fusion characteristic image.
Further, during super-resolution reconstruction processing, the constructed super-resolution reconstruction network comprises a plurality of encoders and decoders, and each encoder is sequentially defined as a 1 st-Nth-level encoder and an Nth-1 st-level decoder according to the forward propagation direction, wherein the value of N is the same as the number of convolutional block levels included in the convolutional neural network for feature information extraction processing; the first-level encoder and the second-level encoder are connected through a splicing layer, and from the 2 nd-level encoder, a down-sampling module and the splicing layer are sequentially connected behind each encoder (namely the down-sampling module and the splicing layer are sequentially connected between the Nth-level encoder and the decoder); the input of the splicing layers also comprises specified-level fusion characteristic images (the resolution of the fusion characteristic images input by each splicing layer is sequentially reduced according to the forward propagation direction), and the characteristic images extracted by the encoder and the input fusion characteristic images are subjected to fusion processing in a dimension splicing mode; the adjacent decoders are sequentially connected with the sampling module and the normalization layer, and the r-level encoder is also accessed into the r-1-level decoder in a jump connection mode (namely the output of the r-level encoder is superposed with the output of the normalization layer accessed into the r-level decoder and then input into the r-1-level decoder), wherein r is more than 1 and less than or equal to N; the 1 st-level decoder is connected to a reconstruction layer, the reconstruction layer is used for reconstructing the characteristic image output by the 1 st-level decoder to obtain a super-resolution reconstruction residual image, and finally the super-resolution reconstruction residual image and the low-resolution input image are superposed to obtain a final super-resolution reconstruction image. Namely, in the invention, the output of each level of encoder is provided with a splicing layer for splicing and fusing the characteristic graphs; except the first-stage encoder, each stage of encoder is provided with a down-sampling layer for down-sampling the feature image and adjusting the size; each level of decoder, except the first level of decoder, is configured with an upsampling layer and a normalization layer.
Further, when the high resolution reference image is down-sampled, a pyramid down-sampling method may be adopted. And in the up-sampling process, a simple interpolation algorithm can be adopted, and the up-sampling and the preliminary super-resolution reconstruction can be carried out by adopting the existing image super-resolution reconstruction method based on a single image.
Furthermore, when image information fusion processing is carried out based on the characteristic diagram, two characteristic sub-blocks to be matched are regarded as vectors, namely, the image information of the characteristic sub-blocks is vectorized, and then the similarity between the two vectors is measured based on the vector similarity, wherein the vector similarity can be obtained by weighting and summing the vector cosine distance and the vector Manhattan distance which are respectively subjected to standardization processing. The greater the vector similarity, the greater the similarity of the two feature sub-blocks. And for a given characteristic sub-block, when the vector similarity reaches the maximum value, the corresponding candidate characteristic sub-block is the optimal matching characteristic sub-block of the given characteristic sub-block.
Further, when image information fusion processing is performed based on the feature map, the low-resolution input feature map and the recombined feature image (obtained by performing feature map recombination on the image position of each sub-block based on the low-resolution input feature map and the second optimal matching feature sub-block) which are respectively subjected to the standardization processing are linearly fused, and the linear fusion result is standardized to obtain a fusion feature image. Namely, the fusion processing of the image information is realized based on the whole migration of the data distribution space.
Furthermore, when the image information fusion processing is performed based on the feature map, the image information fusion processing can be realized based on a linear guide filtering mode, wherein the low-resolution input feature map is used as input, the recombined feature image is used as a guide template, and the output of the linear guide filtering is used as a fusion feature image. The fused feature image is generally similar to the low-resolution input feature map, but it incorporates key feature information of the guide template (i.e., the re-binned feature image).
In summary, due to the adoption of the technical scheme, the invention has the beneficial effects that:
compared with the existing image fusion and super-resolution method, the method adopts the characteristic information extracted by the convolutional neural network to replace the pixel information for image block matching, and realizes information matching, transfer and fusion among images with different resolutions in a characteristic domain. The method can completely and accurately transfer and fuse the high-resolution reference image information to the low-resolution input image, and remarkably improves the super-resolution reconstruction effect of the low-resolution input image.
The invention avoids various defects of the traditional image information matching and fusion based on pixels and the image super-resolution method based on multiple images, solves the problem of image matching across resolutions, and is not limited by the size and the number of images.
Drawings
FIG. 1 is a process diagram illustrating a reconstruction process according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a pretreatment process according to an embodiment;
FIG. 3 is a schematic diagram of a feature extraction network according to an embodiment;
FIG. 4 is a flow diagram illustrating a feature matching and transfer process, in accordance with an exemplary embodiment;
FIG. 5 is a flow diagram illustrating feature fusion, in accordance with an embodiment;
fig. 6 is a schematic diagram of an optimized best matching feature search method according to an embodiment.
Fig. 7 is a schematic diagram of a super-resolution reconstruction process according to an embodiment, where fig. 7(a) is a diagram of a super-resolution reconstruction network structure and fig. 7(b) is a schematic diagram of an encoder/decoder structure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings.
According to the image information fusion and super-resolution reconstruction method based on feature processing, the feature information of the image is adopted to carry out information matching and fusion instead of the pixel information, rich detail texture information carried by a high-resolution reference image can be fully utilized, and the super-resolution reconstruction quality of the low-resolution input image is effectively improved. Referring to fig. 1, the reconstruction method of the present invention requires 1 low resolution input image and at least 1 high resolution input image as inputs to the reconstruction method. Firstly, image preprocessing is carried out to obtain a low-resolution reference image and a preprocessed low-resolution input image. And secondly, performing feature extraction processing on the low-resolution input image, the high-resolution reference image and the low-resolution reference image to obtain respective corresponding feature images. And then carrying out characteristic information matching, transferring and fusing on the high-resolution reference image and the low-resolution input image to obtain a fused characteristic image. And finally, performing super-resolution reconstruction processing on the low-resolution input image and the fusion characteristic image to obtain a final super-resolution reconstruction image.
Referring to fig. 2, in the present embodiment, the manner of performing image preprocessing based on the high-resolution reference image is specifically as follows: and performing Gaussian pyramid downsampling on the high-resolution reference image to obtain a low-resolution reference image. Then, the low-resolution reference image and the low-resolution input image are respectively subjected to bicubic spline interpolation upsampling, and the physical size is enlarged. The down-sampling multiple and the up-sampling multiple of the high-resolution reference image are reciprocal, so that the consistency of the physical size of the up-sampled low-resolution reference image and the physical size of the up-sampled high-resolution reference image are guaranteed. At the same time, the up-sampled multiples of the low resolution reference image and the low resolution input image remain the same, making them similar in pixel sharpness.
Optionally, during image preprocessing, an existing mature single-image-based super-resolution method may be used to perform an upsampling operation, so that preliminary single-image super-resolution reconstruction may be implemented while upsampling. The method of upsampling is user-defined. The super-resolution reconstruction method based on the single image can ensure that the final super-resolution reconstruction effect is not lower than that of the super-resolution reconstruction method based on the single image under the worst condition by introducing the existing super-resolution network based on the single image. Meanwhile, from the optimization point of view, the process helps the super-resolution reconstruction network to converge more quickly during training.
In this embodiment, the feature extraction processing on the image is implemented by a neural network, and the constructed feature extraction network is essentially a convolutional neural network, and includes several stages of convolution blocks, the convolution blocks are connected by pooling layers, each stage of convolution block includes a certain number of convolutions and nonlinear activation processing, as shown in fig. 3, where conv denotes a convolution layer, ReLU denotes a ReLU activation layer, and poolling denotes a pooling layer. The pooling multiple of the pooling layer is 2, and convi _ j and relui _ j are adopted to distinguish different convolution layers and ReLU activation layers, wherein i represents the number of stages, and j represents the corresponding number of layers. In addition, in the feature extraction process, in order to ensure that the sizes of the input and output images of each layer are consistent, all the convolution layers need to be subjected to edge-to-edge 0 compensation.
In the specific embodiment, three layers of outputs of relu1_1, relu2_1 and relu3_1 are selected as feature images extracted by the feature extraction network, and are named as a primary feature image relu1_1, a secondary feature image relu2_1 and a tertiary feature image relu3_1 respectively. The three levels of features represent three different levels of image features respectively. Assuming that the input image has a size H × W × 3(RGB three-channel image), the output three-level feature image has a physical size H × W × C1，(H/2)*(W/2)*C2And (H/4) (W/4) () C3. Wherein C is1，C2And C3Respectively representing the number of channels of the feature images of different levels.
Optionally, in the process of implementing the feature extraction network shown in fig. 3, the present invention supports importing parameters in an external image recognition convolutional neural network as parameters of the feature extraction network. Existing image recognition convolutional neural networks (e.g., VGG networks) typically include an image feature extraction component. After a long time of application, the parameters of the components are relatively stable and mature, and can be directly used without worrying about the problem of unstable image feature extraction. Compared with the method for re-customizing the feature extraction network, the method has the advantages that the workload can be effectively reduced by importing the parameters of the mature network, and the final super-resolution reconstruction result cannot be obviously adversely affected.
Referring to fig. 4, in the process of feature matching and transfer, in the present embodiment, first, the feature map R0_ F of the high-resolution reference image, the feature map R1_ F of the low-resolution reference image, and the feature map LR _ F of the low-resolution input image need to be respectively split to obtain a plurality of feature sub-blocks. Each feature sub-block is considered a vector, all of the same physical size and possibly spatially coincident with each other. Given the ith feature sub-block LR in the feature image LR _ FiFor each feature sub-block in the feature image R1_ F, the invention defines a feature matching algorithm, and two feature sub-blocks are measured by calculating the vector similarity VSThe similarity between them. The calculation formula of VS is:
among them, LRiRepresents the ith feature sub-block, R1, in the feature image LR _ FjRepresents the jth feature sub-block in the feature image R1_ F, and w represents a weighting between 0 and 1. IP represents the inner product of the vector, and MD represents the manhattan distance (also called city distance) of the vector. | denotes the modulo length of the vector, |normIndicating a normalization process. In calculating IP, R1jRequired as unit modulo length vector
This eliminates R1jThe result of the computation of the vector inner product IP can be mathematically equivalent to the cosine distance of the vector. Meanwhile, in order to overcome the inconsistency of the IP and the MD in measurement, the standardization operation is needed after the numerical values of the IP and the MD are calculated. In particular, for a given LRiAll candidate feature sub-blocks R1 in the feature image R1_ F are calculatedjIP value and MD value of (d). And respectively normalizing the IP values and the MD values of all candidate feature subblocks, and eliminating the difference in the metric after normalization. Then weighted and the final VS value is obtained.
It can be seen from the VS formula that the larger the VS value, the higher the similarity of the 2 feature sub-blocks. Note j*Is the i-th feature sub-block LR in the feature image R1_ F and in the given feature image LR _ FiThe feature sub-block index with the highest similarity,
representing the best matching feature sub-block in R1_ F. When VS(i,j)When the maximum is obtained, the corresponding j is j*. The mathematical expression is:
after solving to obtain j*After that, j' th on R1_ F*Individual character sub-block
Has already been determined, here
Referred to as the first best matching feature sub-block. Then, the spatial position mapping relationship between R0_ F and R1_ F is utilized to obtain the jth position in R0_ F*Individual character sub-block
Is the ith characteristic sub-block LR in LR _ FiBest matching feature sub-block of, where
Referred to as the second best matching feature sub-block. From the information matching perspective, the second best matching feature sub-block
The contained information is exactly the ith characteristic sub-block LR in LR _ FiAs required.
The last step is feature transfer, the second best matching feature sub-block
Will fill in the specific locations of the recombined feature image RC _ F. The size of the recombined feature image is the same as the size of LR _ F,
the filling position in RC _ F is the i-th characteristic sub-block LR in LR _ FiRelative position in LR _ F.
For each feature sub-block in LR _ F, the second best matching feature sub-block in R0_ F can always be found by the above matching method. And then filling the second optimal matching feature sub-block into the corresponding position of the recombined feature image according to the given relative position of the feature sub-block. Since different feature sub-blocks may coincide with each other during the splitting process of the feature sub-blocks. Therefore, the second best matching feature sub-blocks are also overlapped when filling the regrouped feature image. Further, when multiple feature sub-blocks spatially overlap each other, this conflict can be resolved by solving the average of the overlapping elements at spatial positions.
Referring to fig. 5, in the process of feature fusion, 4 variables need to be calculated first: the mean and variance of the reconstructed feature image RC _ F and the mean and variance of the low-resolution input feature image LR _ F. After the 4 variables are obtained, calculating a fusion feature image M _ F through a feature fusion formula:
M_F＝||μ*||LR_F||norm+(1-μ)*||RC_F||norm||norm
wherein | | xi | purplenormFor the normalization operation, the mean and variance of the input data need to be calculated when performing the normalization. μ is a weighted weight between 0 and 1.
Optionally, in the process of feature fusion, a feature fusion mode based on guided filtering may also be adopted, and the specific implementation process is as follows:
let the low resolution input feature map be p, the reconstructed feature map be I, and give the window radius r and the regularization parameter epsilon. The window radius r represents the area where the mean is calculated as an r x r area.
Calculating mean image mean of feature map pp＝fmean(p), mean image mean of the recombined feature map II＝fmean(I)；
Squaring the pixel value of each pixel point of the recombined feature map I to obtain an image I2And calculating its mean image corrI＝fmean(I.) the recombined feature map is multiplied by the pixel values of the same positions of the I and the p feature maps to obtain an image Ip＝fmean(I. p); where a · B denotes multiplying the pixel values of the same pixel point position in images a and B.
Computing variance image var of reorganized feature map II＝corrI-meanI.*meanIAnd a covariance image cov of the recombined feature map I and the feature map pIp＝corrIp-meanI.*meanp；
Cov according to formula aIp/(varI+ epsilon) calculating the parameter a, and mean according to the formula bp-a.*meanICalculating the parameter b to obtain the mean value mean of the parameters a and ba、meanbFinally according to the formula q meana.*I+meanbAnd obtaining a fusion characteristic map q.
Optionally, in order to speed up the matching process, the feature matching process only matches the tertiary feature image relu31 in the present invention. For the primary characteristic image and the secondary characteristic image, matching can be performed according to the matching result of the tertiary characteristic image and the spatial correspondence. When two feature sub-blocks in the tertiary feature image are successfully matched, the matching result can be mapped to the primary feature image and the secondary feature image.
Optionally, in the process of calculating the best matching feature sub-block, on the premise that the similarity between the low-resolution input image and the high-resolution reference image is high enough (greater than or equal to a specified similarity threshold), the search range may be constrained by means of the matching position information of the adjacent feature sub-blocks, which is helpful to greatly reduce the calculation amount in the matching process and increase the operation speed. Referring to fig. 6, the optimized optimal matching feature search process of the present invention specifically includes: for a certain feature sub-block X on LR _ F, the first best matching feature sub-block on R1_ F is XO. For the adjacent feature sub-block Y of X, the center position of the last first best matching feature sub-block XO may be relatively shifted (the specific shift amount is set according to the specific processing requirement), so as to obtain the search anchor point. A smaller area is then determined by the location of the anchor point to search for the first best matching sub-block Y0 of Y. The search area is usually set to be rectangular, and the specific range of the search area is determined based on the distance from the configured anchor point to each side of the rectangular search area. Experiments show that in most cases, the best matching result obtained by the fast search method is consistent with the result obtained by the complete full-graph search method described in fig. 4, and the final reconstructed effect graph of the 2 matching schemes has no significant difference from the final reconstruction perspective.
In this embodiment, the super-resolution reconstruction processing is implemented by a convolutional network, and referring to fig. 7(a), the employed super-resolution reconstruction convolutional neural network is an encoding-decoding structure as a whole. The first half of the network has 3 encoders, wherein the second and third encoders are followed by a downsampling module and a Batch Normalization layer. The down-sampling module is a convolution layer with a step size of 2, which can adjust the length and width of the feature image output by the encoder to 1/2, and the length and width of the feature image are consistent with the physical size of the fused feature image spliced from the outside. And on the splicing layer, performing dimension splicing on the feature image of the original network and the fusion feature image in the last dimension. For example, assume the original network feature image size is B H W C1The fused feature image size is BxH xW C2Then the size of the feature image after stitching is B × H × W (C)1+C2) Where B, H, C denotes the number of images per batch, the height of the images and the width of the images, respectively, C1And C2Representing the number of characteristic image channels. The latter half of the network corresponds to 3 decoders, wherein the second decoder and the third decoder are followed by an up-sampling module and a Batch Normalization layer. The up-sampling module is a deconvolution layer with the step length of 2 and is used for adjusting the length and the width of the characteristic image output by the decoder to 2 times of the original length and the width. 3 long-distance jump connection layers exist in the network, the last layer of the network is a reconstruction layer, an input characteristic image is reconstructed to obtain a super-resolution reconstruction residual image, the reconstruction layer can be set as a convolution layer with the step length of 1, the number of output channels is adjusted based on actual conditions, for example, for an RGB image, the number of the output channels is set to be 3; for a single Y-channel image (called a luminance image), the number of output channels is 1; and finally, superposing the super-resolution reconstruction residual image and the original input image to obtain a final super-resolution reconstruction result. In the super-resolution reconstruction process, all the convolution layers except the down-sampling module need to be processed in order to ensure that the input and output images of each layer have consistent sizeThe edge padding process is performed.
Referring to fig. 7(b), the encoder/decoder internal structure is: each encoder and decoder is internally composed of 5 convolutional layers conv and 1 Batch Normalization layer. The activation function for each convolutional layer is the pReLU function. The convolution layers are connected with each other in a dense connection mode, and meanwhile, a jump connection layer is arranged, and the output of the first convolution layer and the output of the last convolution layer are added to obtain the final output. Dense connections can enable image features to be multiplexed, and network performance is improved. And the jump connection enables the network to learn the residual error between the input and the expected output in the training process instead of the expected output, which can effectively reduce the network training burden and also has a positive effect on the network performance improvement.
In this embodiment, the training and the application (i.e. super-resolution reconstruction) of the super-resolution reconstruction network are respectively as follows:
in the training process, training data is first prepared and divided into a training set and a test set. The training set is used for training and updating parameters of the super-resolution reconstruction network, and the test set is used for evaluating network performance. In the present embodiment, the feature extraction network imports part of the parameters of the external network, and therefore does not perform updating. In order to accelerate the training speed, an off-line feature processing process is firstly carried out, and the data of the fused feature images are stored as intermediate data. And then importing the data when training the super-resolution reconstruction network for updating network parameters and storing a network model. And after the training is finished, the stored parameters of the super-resolution reconstruction network can be exported.
In the application stage, image preprocessing and feature processing are firstly performed on a given high-resolution reference image and a given low-resolution input image to obtain fused feature image data, and then the fused feature image data and the fused feature image data are sent to a super-resolution reconstruction network for super-resolution reconstruction, so that a final result is obtained.
While the invention has been described with reference to specific embodiments, any feature disclosed in this specification may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise; all of the disclosed features, or all of the method or process steps, may be combined in any combination, except mutually exclusive features and/or steps.
Claims (10)
1. An image information fusion and super-resolution reconstruction method based on feature processing is characterized by comprising the following steps:
image preprocessing is performed on the high resolution reference image and the low resolution input image: down-sampling the high-resolution reference image to obtain a low-resolution reference image matched with the definition of the low-resolution input image; and performing the same upsampling on the low-resolution input image and the low-resolution reference image;
extracting feature information of the high-resolution reference image, the up-sampled low-resolution reference image and the low-resolution input image to obtain a high-resolution reference feature map, a low-resolution reference feature map and a low-resolution input feature map; and carrying out image information matching, transferring and fusing on the obtained characteristic diagram:
carrying out blocking processing on the high-resolution reference characteristic diagram, the low-resolution reference characteristic diagram and the low-resolution input characteristic diagram;
traversing each feature sub-block in the low-resolution input feature map, searching a first optimal matching feature sub-block in the low-resolution reference feature map, and determining a second optimal matching feature sub-block of the current sub-block in the high-resolution reference feature map based on the image position of the first optimal matching feature sub-block based on the spatial mapping relation between the high-resolution reference feature map and the low-resolution reference feature map;
performing feature map reorganization processing on the basis of the image position of each sub-block of the low-resolution input feature map and the second optimal matching feature sub-block to obtain a reorganized feature image;
performing image information fusion processing based on the recombined feature map to obtain a fusion feature image;
and performing super-resolution reconstruction on the low-resolution input image based on the fusion characteristic image to obtain a super-resolution reconstruction image.
2. The method of claim 1, wherein the super-resolution reconstruction process is performed by a super-resolution reconstruction network based on an encoding and decoding structure, wherein an encoding portion of the super-resolution reconstruction network is used for feature extraction of the low-resolution input image and fusing the extracted feature image with the fused feature image by means of dimension stitching; the decoding part is used for reconstructing the fused characteristic image and outputting a super-resolution reconstructed image.
3. The method according to claim 1, wherein a feature extraction network based on a convolutional neural network is used for performing feature information extraction processing on the high-resolution reference image, the up-sampled low-resolution reference image and the low-resolution input image, the convolutional neural network comprises a plurality of stages of convolution blocks which are connected in sequence, the convolution blocks are connected through a pooling layer, each stage of convolution block comprises a plurality of sub-layers which are connected in sequence, each sub-layer is composed of a convolution layer and a nonlinear activation layer which are connected in sequence, and one nonlinear activation layer is appointed from each stage of convolution block to be output as a feature map of the current stage, so that a multi-stage high-resolution reference feature map, a multi-stage low-resolution reference feature map and a multi-stage low-resolution input feature map are obtained; and carrying out image information matching, transferring and fusing on the characteristic images at the same level to obtain a multi-level fused characteristic image.
4. The method according to claim 2, wherein the super-resolution reconstruction network comprises a plurality of encoders and decoders, and in the forward propagation direction, each encoder is defined as a level 1 to a level N encoder, and a level N to a level 1 decoder, wherein the value of N is the same as the number of convolutional block levels included in the convolutional neural network for feature information extraction processing; the first-level encoder and the second-level encoder are connected through a splicing layer, and from the 2 nd-level encoder, a down-sampling module and the splicing layer are sequentially connected behind each encoder; the input of the splicing layer also comprises a fusion characteristic image of a designated level, and the characteristic image extracted by the encoder and the input fusion characteristic image are subjected to fusion processing in a dimension splicing mode; the adjacent decoders are sequentially connected with a sampling module and a normalization layer, and the r-level encoder is also connected into the r-1-level decoder in a jump connection mode, wherein r is more than 1 and less than or equal to N; the 1 st-level decoder is connected to a reconstruction layer, the reconstruction layer is used for reconstructing the characteristic image output by the 1 st-level decoder to obtain a super-resolution reconstruction residual image, and finally the super-resolution reconstruction residual image and the low-resolution input image are superposed to obtain a final super-resolution reconstruction image.
5. The method according to claim 3, wherein for the multi-level high-resolution reference feature map, the low-resolution reference feature map, and the low-resolution input feature map, when performing the feature sub-block matching processing on the feature map of the same level, the feature block matching processing of the first and second optimally matched feature sub-blocks is performed only on the last-level feature map of the convolutional neural network for the feature information extraction processing, and for the feature block matching processing between the feature maps of the remaining levels, the corresponding matching results are obtained directly based on a spatial mapping relationship of the result of the feature block matching processing between the last-level feature maps.
6. The method according to claim 1, wherein when performing image information fusion processing based on the feature map, two feature sub-blocks to be matched are used as vectors, the degree of matching of the two feature sub-blocks is used as based on vector similarity, and a first best matching feature sub-block of the feature sub-blocks in the low-resolution input feature map is searched based on the degree of matching between the feature sub-blocks.
7. The method of claim 6, wherein the vector similarity is: and respectively carrying out standardization processing on the vector cosine distance and the vector Manhattan distance of the two vectors, and then carrying out weighted summation to obtain the vector Manhattan distance.
8. The method according to claim 1, wherein when the image information fusion processing is performed based on the feature map, feature map reorganization is performed based on the image position of each sub-block of the low-resolution input feature map and the second best matching feature sub-block to obtain a reorganized feature image, the low-resolution input feature map and the reorganized feature image which are respectively subjected to the normalization processing are linearly fused, and the linear fusion result is subjected to the normalization processing to obtain a fusion feature image.
9. The method according to claim 1, wherein the image information fusion processing is performed based on a feature map, and the method is implemented based on a linear guide filtering method, wherein the low-resolution input feature map is used as an input, the reconstructed feature image is used as a guide template, and an output of the linear guide filtering is used as a fusion feature image.
10. The method of claim 1, wherein in searching for a first best matching feature sub-block of the feature sub-blocks in the low resolution input feature map, if the similarity between the low resolution input image and the high resolution reference image is greater than or equal to a specified similarity threshold, constraining a search range of the first best matching feature sub-block based on matching location information of neighboring feature sub-blocks:
defining a certain feature sub-block in the low-resolution input feature map as a feature sub-block X, wherein a first optimal matching feature sub-block in the low-resolution reference feature map is X0;
for the adjacent feature sub-block Y of the feature sub-block X, carrying out relative offset on the central position of X0 to obtain a search anchor point;
and determining the search range of the first optimal matching feature sub-block of the feature sub-block Y based on the distance between the configured search anchor point and each boundary of the search range of the first optimal matching feature sub-block.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011528460.3A CN112598575B (en) | 2020-12-22 | 2020-12-22 | Image information fusion and super-resolution reconstruction method based on feature processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011528460.3A CN112598575B (en) | 2020-12-22 | 2020-12-22 | Image information fusion and super-resolution reconstruction method based on feature processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112598575A CN112598575A (en) | 2021-04-02 |
| CN112598575B true CN112598575B (en) | 2022-05-03 |
Family
ID=75200199
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011528460.3A Active CN112598575B (en) | 2020-12-22 | 2020-12-22 | Image information fusion and super-resolution reconstruction method based on feature processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112598575B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114418853B (en) * | 2022-01-21 | 2022-09-20 | 杭州碧游信息技术有限公司 | Image super-resolution optimization method, medium and equipment based on similar image retrieval |
| CN114723986A (en) * | 2022-03-16 | 2022-07-08 | 平安科技（深圳）有限公司 | Text image matching method, device, equipment and storage medium |
| CN114760435A (en) * | 2022-06-13 | 2022-07-15 | 深圳达慧信息技术有限公司 | Conference relaying method, device, equipment and storage medium based on image processing |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016019484A1 (en) * | 2014-08-08 | 2016-02-11 | Xiaoou Tang | An apparatus and a method for providing super-resolution of a low-resolution image |
| CN106228512A (en) * | 2016-07-19 | 2016-12-14 | 北京工业大学 | Based on learning rate adaptive convolutional neural networks image super-resolution rebuilding method |
| CN106952228A (en) * | 2017-03-10 | 2017-07-14 | 北京工业大学 | The super resolution ratio reconstruction method of single image based on the non local self-similarity of image |
| CN111667412A (en) * | 2020-06-16 | 2020-09-15 | 中国矿业大学 | Method and device for reconstructing image super-resolution based on cross learning network |
| CN111915484A (en) * | 2020-07-06 | 2020-11-10 | 天津大学 | Reference image guiding super-resolution method based on dense matching and self-adaptive fusion |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10339633B2 (en) * | 2015-11-04 | 2019-07-02 | Peking University Shenzhen Graduate School | Method and device for super-resolution image reconstruction based on dictionary matching |
-
2020
- 2020-12-22 CN CN202011528460.3A patent/CN112598575B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016019484A1 (en) * | 2014-08-08 | 2016-02-11 | Xiaoou Tang | An apparatus and a method for providing super-resolution of a low-resolution image |
| CN106228512A (en) * | 2016-07-19 | 2016-12-14 | 北京工业大学 | Based on learning rate adaptive convolutional neural networks image super-resolution rebuilding method |
| CN106952228A (en) * | 2017-03-10 | 2017-07-14 | 北京工业大学 | The super resolution ratio reconstruction method of single image based on the non local self-similarity of image |
| CN111667412A (en) * | 2020-06-16 | 2020-09-15 | 中国矿业大学 | Method and device for reconstructing image super-resolution based on cross learning network |
| CN111915484A (en) * | 2020-07-06 | 2020-11-10 | 天津大学 | Reference image guiding super-resolution method based on dense matching and self-adaptive fusion |
Non-Patent Citations (6)
| Title |
|---|
| Learning a Deep Convolutional Network for Image Super-Resolution;Chao Dong等;《European conference on computer vision》;20141231;第184-199页 * |
| Super-Resolution Through Neighbor Embedding;Hong Chang等;《Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition》;20041231;第1-8页 * |
| 图像超分辨率卷积神经网络加速算法;刘超 等;《国防科技大学学报》;20190430;第41卷(第2期);第91-97页 * |
| 基于CNN迁移特征融合与池化的高分辨率遥感图像检索研究;葛芸;《中国博士学位论文全文数据库 工程科技Ⅱ辑》;20190415;第C028-6页 * |
| 基于卷积神经网络的视频超分辨技术研究;敬琳萍;《中国优秀硕士学位论文全文数据库 信息科技辑》;20191215;第I138-607页 * |
| 基于稀疏表示的图像超分辨率重建方法研究;李炜;《中国优秀硕士学位论文全文数据库 信息科技辑》;20181015;第I138-614页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112598575A (en) | 2021-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112598575B (en) | Image information fusion and super-resolution reconstruction method based on feature processing | |
| CN109509152B (en) | Image super-resolution reconstruction method for generating countermeasure network based on feature fusion | |
| CN113538246B (en) | Remote sensing image super-resolution reconstruction method based on unsupervised multi-stage fusion network | |
| CN113012172A (en) | AS-UNet-based medical image segmentation method and system | |
| CN108764250B (en) | Method for extracting essential image by using convolutional neural network | |
| CN113538234A (en) | Remote sensing image super-resolution reconstruction method based on lightweight generation model | |
| CN114283120B (en) | Domain-adaptive-based end-to-end multisource heterogeneous remote sensing image change detection method | |
| CN109598676A (en) | A kind of single image super-resolution method based on Hadamard transform | |
| CN113744136A (en) | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion | |
| CN113256494B (en) | Text image super-resolution method | |
| CN113436237B (en) | High-efficient measurement system of complicated curved surface based on gaussian process migration learning | |
| CN112686830B (en) | Super-resolution method of single depth map based on image decomposition | |
| CN114092824A (en) | Remote sensing image road segmentation method combining intensive attention and parallel up-sampling | |
| CN117974693B (en) | Image segmentation method, device, computer equipment and storage medium | |
| CN115205527A (en) | Remote sensing image bidirectional semantic segmentation method based on domain adaptation and super-resolution | |
| CN116597203A (en) | Knowledge distillation-based anomaly detection method for asymmetric self-encoder | |
| CN117079105A (en) | Remote sensing image spatial spectrum fusion method and device, electronic equipment and storage medium | |
| CN117593199A (en) | Double-flow remote sensing image fusion method based on Gaussian prior distribution self-attention | |
| CN113724307A (en) | Image registration method and device based on characteristic self-calibration network and related components | |
| CN115035377B (en) | Significance detection network system based on double-flow coding and interactive decoding | |
| CN116152263A (en) | CM-MLP network-based medical image segmentation method | |
| Yang | Super resolution using dual path connections | |
| Zhou et al. | Structure and texture preserving network for real-world image super-resolution | |
| CN115496910B (en) | Point cloud semantic segmentation method based on full-connected graph coding and double-expansion residual error | |
| CN118395361B (en) | Bridge cable acoustic emission broken wire signal identification method and system under unbalanced data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |