CN107133921B - Image super-resolution reconstruction method and system based on multilevel neighborhood embedding - Google Patents
Image super-resolution reconstruction method and system based on multilevel neighborhood embedding Download PDFInfo
- Publication number
- CN107133921B CN107133921B CN201610108841.3A CN201610108841A CN107133921B CN 107133921 B CN107133921 B CN 107133921B CN 201610108841 A CN201610108841 A CN 201610108841A CN 107133921 B CN107133921 B CN 107133921B
- Authority
- CN
- China
- Prior art keywords
- image
- pass
- band
- images
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 67
- 238000012545 processing Methods 0.000 claims abstract description 39
- 230000009466 transformation Effects 0.000 claims abstract description 16
- 238000005457 optimization Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 5
- 230000000694 effects Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000001914 filtration Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000005352 clarification Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G06T5/73—
Abstract
The invention provides an image super-resolution reconstruction method based on multi-level neighborhood embedding and an image super-resolution reconstruction system based on multi-level neighborhood embedding, wherein the method comprises the following steps: carrying out controllable pyramid decomposition operation on an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions; reconstructing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions respectively according to a neighborhood embedding algorithm to obtain corresponding high-resolution sub-band images; and performing inverse transformation processing of the controllable pyramid decomposition operation on the plurality of high-resolution sub-band images to generate a high-resolution image. By the technical scheme, the problem that texture details of the image are smoothed due to direct reconstruction of all high-frequency components of the image can be effectively solved, and the accuracy of super-resolution reconstruction of the image is improved.
Description
Technical Field
The invention relates to the technical field of computer image processing, in particular to an image super-resolution reconstruction method based on multi-level neighborhood embedding and an image super-resolution reconstruction system based on multi-level neighborhood embedding.
Background
Image super-resolution reconstruction is to reconstruct a high resolution image from a single frame of a low resolution image or sequence of low resolution images in order to overcome the limitations of the imaging process equipment or technique. At present, image super-resolution reconstruction algorithms can be classified into three categories: interpolation-based algorithms, reconstruction-based algorithms and learning-based algorithms.
Specifically, the method comprises the following steps: (1) the Interpolation-based algorithm adopts a linear or non-linear algorithm, and estimates an unknown pixel value by using a known pixel point, such as NEDI (New Edge-Directed Interpolation) and SAI (Soft-precision Adaptive Interpolation). (2) The reconstruction-based algorithm is to use a maximum a posteriori likelihood model to characterize natural images using various regularization terms as constraints. (3) The learning-based algorithm is to build the relationship between low-resolution image blocks and high-resolution image blocks depending on an external data set, and researchers in the field propose different models to describe the relationship, including describing by using a markov random field model, wherein each low-resolution image block corresponds to several high-resolution candidate image blocks, and the characteristic conforms to the structural construction of the markov random field model and can be solved by image segmentation or belief propagation, but the algorithm has high computational complexity.
In addition, Yang et al (an author of 2010TIP Image super-resolution video representation) proposes an Image super-resolution algorithm based on sparse representation, in which dictionaries of high and low resolution images are trained, an input low resolution Image is sparsely represented by a low resolution dictionary, and then the sparse coefficient is multiplied by a corresponding high resolution dictionary to obtain a high resolution Image. And, based on the assumption that high and low resolution image blocks share similar linear geometric manifolds, Chang et al (authors of Hong Chang et al, 2004CVPR Super-resolution through neighbor embedding) proposed a neighborhood embedding algorithm.
However, the existing image super-resolution algorithms directly reconstruct the spatial difference part of the image, i.e. the high-frequency component of the image, however, the structural features of the image may be embodied in the frequency domains in different directions, and particularly in the case of rich image texture, directly reconstructing all the high-frequency components of the image may cause the texture details of the image to be directly smoothed.
Therefore, how to solve the above problems and improve the accuracy of super-resolution image reconstruction become a technical problem to be solved urgently.
Disclosure of Invention
Based on the technical problems, the invention provides a new technical scheme, which can effectively solve the problem that the texture details of the image are smoothed due to the fact that all high-frequency components of the image are directly reconstructed, and therefore the accuracy of image super-resolution reconstruction is improved.
In view of the above, the first aspect of the present invention provides an image super-resolution reconstruction method based on multi-level neighborhood embedding, including: carrying out controllable pyramid decomposition operation on an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions; reconstructing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions respectively according to a neighborhood embedding algorithm to obtain corresponding high-resolution sub-band images; and performing inverse transformation processing of the controllable pyramid decomposition operation on the plurality of high-resolution sub-band images to generate a high-resolution image.
In the technical scheme, a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, which are obtained by subjecting an image to be processed (such as a single-frame low-resolution image or a low-resolution image sequence) to controllable pyramid decomposition operation, are reconstructed by using a neighborhood embedding algorithm, and then the high-resolution image is generated by inverse transformation of the controllable pyramid decomposition operation, so that the problem that texture details of the image are smoothed due to direct reconstruction of all high-frequency components of the image in the existing image super-resolution reconstruction method is effectively solved, and the accuracy of image super-resolution reconstruction is improved.
In the foregoing technical solution, preferably, the performing a controllable pyramid decomposition operation on the image to be processed to obtain a high-pass image, a low-pass residual image, and band-pass subband images in multiple directions includes: calculating a group of multidirectional controllable band-pass filters of the image to be processed according to a preset decomposition formulaTo obtain the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions, wherein the preset decomposition formula is as follows:
wherein, XtRepresenting the image to be processed in a manner such that,
representing band-pass sub-bands, X, in said plurality of directionst 0Representing said high-pass image, Xt N+1Representing said low-pass residual image, F: (·) And F-1(·) Respectively representing the Fourier transform and the inverse Fourier transform, f (theta)i) Representative direction is θiControllable band-pass filter of giRepresenting a high pass filter or a low pass filter calculated from a controllable band pass filter.
In the technical scheme, a group of multidirectional controllable band-pass filters are used for filtering an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, so that multiple image levels are obtained, and the images of all the image levels are subjected to corresponding neighborhood embedding processing.
In any of the above technical solutions, preferably, the reconstructing the band-pass subband images, the high-pass images, and the low-pass residual images in the multiple directions according to a neighborhood embedding algorithm to obtain corresponding high-resolution subband images includes: respectively decomposing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions into corresponding image blocks; searching K adjacent neighborhood image blocks in a dictionary database of a corresponding image layer by utilizing a K-NN algorithm for each image block to obtain K reconstruction coefficients; and obtaining the corresponding high-resolution sub-band image according to the K reconstruction coefficients and the K adjacent domain image blocks.
In the technical scheme, when reconstructing the image of each image level obtained by decomposition according to a neighborhood embedding algorithm, firstly, the corresponding image needs to be decomposed into a plurality of image blocks, and K-NN algorithm (K-Nearest Neighbor, an algorithm for finding Nearest neighbors) is used for each image block to obtain K neighborhood image blocks (i.e. image blocks with high similarity) Nearest to the image block, so as to estimate corresponding K reconstruction coefficients, and then each neighborhood image block is multiplied by the corresponding reconstruction coefficient to combine the multiplication results, so as to obtain a high-resolution subband image with significantly improved resolution, wherein a dictionary database corresponding to the image level refers to all high-frequency image sets, low-frequency image sets and image sets corresponding to each direction formed after decomposing the image to be processed.
In any of the above technical solutions, preferably, for the band-pass subband images and the high-pass images in the multiple directions, the K neighboring neighborhood image blocks are searched in a dictionary database of a corresponding image hierarchy according to global feature information and local feature information; and for the low-pass residual image, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to the joint local feature information.
In the technical scheme, the reconstruction quality of each image block depends on the accuracy of the found nearest neighbor image block, so that the standard for selecting similar blocks is crucial, and in order to ensure the reconstruction effect of image blocks of different image levels, for a high-pass image and band-pass sub-band images in multiple directions, the image block characteristic information of the image to be processed is introduced as global characteristic information, and the characteristic information represented by the image blocks of the high-pass image and the band-pass sub-band images in multiple directions is used as local characteristic information; while for low-pass residual images, their gradient features cannot be extracted efficiently because their low frequency is smoother, but they are continuous and already contain sufficient image structure information, thus introducing joint local feature information as a measure of their similarity. In addition, the feature information refers to texture feature information of the image and the like so as to ensure the accuracy of super-resolution reconstruction of the image.
In any of the above technical solutions, preferably, the method further includes: and optimizing the high-resolution image by using a non-local mean algorithm.
In the technical scheme, in order to obtain a better reconstruction effect, a non-local mean algorithm is used for carrying out optimization processing on the high-resolution image obtained through inverse transformation processing of the controllable pyramid decomposition operation so as to reduce reconstruction errors among similar image blocks of the high-resolution image.
In a second aspect of the present invention, an image super-resolution reconstruction system based on multi-level neighborhood embedding is provided, including: the decomposition module is used for carrying out controllable pyramid decomposition operation on the image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions; the reconstruction module is used for respectively reconstructing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions, which are obtained by the decomposition module, according to a neighborhood embedding algorithm to obtain corresponding high-resolution sub-band images; and the processing module is used for performing inverse transformation processing of the controllable pyramid decomposition operation on the plurality of high-resolution sub-band images obtained by the processing of the reconstruction module so as to generate a high-resolution image.
In the technical scheme, a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, which are obtained by subjecting an image to be processed (such as a single-frame low-resolution image or a low-resolution image sequence) to controllable pyramid decomposition operation, are reconstructed by using a neighborhood embedding algorithm, and then the high-resolution image is generated by inverse transformation of the controllable pyramid decomposition operation, so that the problem that texture details of the image are smoothed due to direct reconstruction of all high-frequency components of the image in the existing image super-resolution reconstruction method is effectively solved, and the accuracy of image super-resolution reconstruction is improved.
In the foregoing technical solution, preferably, the decomposition module is specifically configured to: calculating the response of the image to be processed to a group of multi-directional controllable band-pass filters according to a preset decomposition formula so as to obtain band-pass sub-band images in multiple directions, the high-pass image and the low-pass residual image, wherein the preset decomposition formula is as follows:
wherein, XtRepresenting the image to be processed in a manner such that,
representing band-pass sub-bands, X, in said plurality of directionst 0Representing said high-pass image, Xt N+1Representing said low-pass residual image, F: (·) And F-1(·) Respectively representing the Fourier transform and the inverse Fourier transform, f (theta)i) Representative direction is θiControllable band-pass filter of giRepresenting a high pass filter or a low pass filter calculated from a controllable band pass filter.
In the technical scheme, a group of multidirectional controllable band-pass filters are used for filtering an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, so that multiple image levels are obtained, and the images of all the image levels are subjected to corresponding neighborhood embedding processing.
In any one of the above technical solutions, preferably, the reconstruction module specifically includes: the decomposition sub-module is used for decomposing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions into corresponding image blocks respectively; the searching submodule is used for searching K adjacent neighborhood image blocks in a dictionary database of a corresponding image layer by utilizing a K-NN algorithm for each image block obtained by decomposing the decomposing submodule so as to obtain K reconstruction coefficients; and the processing submodule is used for obtaining the corresponding high-resolution sub-band image according to the K reconstruction coefficients and the K adjacent domain image blocks obtained by the processing of the searching submodule.
In the technical scheme, when reconstructing the image of each image level obtained by decomposition according to a neighborhood embedding algorithm, firstly, the corresponding image needs to be decomposed into a plurality of image blocks, and K-NN algorithm (K-Nearest Neighbor, an algorithm for finding Nearest neighbors) is used for each image block to obtain K neighborhood image blocks (i.e. image blocks with high similarity) Nearest to the image block, so as to estimate corresponding K reconstruction coefficients, and then each neighborhood image block is multiplied by the corresponding reconstruction coefficient to combine the multiplication results, so as to obtain a high-resolution subband image with significantly improved resolution, wherein a dictionary database corresponding to the image level refers to all high-frequency image sets, low-frequency image sets and image sets corresponding to each direction formed after decomposing the image to be processed.
In any of the above technical solutions, preferably, the search submodule is specifically configured to: for the band-pass sub-band images and the high-pass images in the multiple directions, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to global characteristic information and local characteristic information; and for the low-pass residual image, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to the joint local feature information.
In the technical scheme, the reconstruction quality of each image block depends on the accuracy of the found nearest neighbor image block, so that the standard for selecting similar blocks is crucial, and in order to ensure the reconstruction effect of image blocks of different image levels, for a high-pass image and band-pass sub-band images in multiple directions, the image block characteristic information of the image to be processed is introduced as global characteristic information, and the characteristic information represented by the image blocks of the high-pass image and the band-pass sub-band images in multiple directions is used as local characteristic information; while for low-pass residual images, their gradient features cannot be extracted efficiently because their low frequency is smoother, but they are continuous and already contain sufficient image structure information, thus introducing joint local feature information as a measure of their similarity. In addition, the feature information refers to texture feature information of the image and the like so as to ensure the accuracy of super-resolution reconstruction of the image.
In any of the above technical solutions, preferably, the method further includes: and the optimization module is used for optimizing the high-resolution image obtained by the processing module by using a non-local mean algorithm.
In the technical scheme, in order to obtain a better reconstruction effect, a non-local mean algorithm is used for carrying out optimization processing on the high-resolution image obtained through inverse transformation processing of the controllable pyramid decomposition operation so as to reduce reconstruction errors among similar image blocks of the high-resolution image.
By the technical scheme, the problem that texture details of the image are smoothed due to the fact that all high-frequency components of the image are directly reconstructed can be effectively solved, and therefore the accuracy of super-resolution reconstruction of the image is improved.
Drawings
FIG. 1 is a flow chart of a super-resolution image reconstruction method based on multi-level neighborhood embedding according to an embodiment of the present invention;
FIG. 2 shows a decomposition result diagram of a low resolution image according to an embodiment of the invention;
FIG. 3 is a block diagram of a multi-level neighborhood embedding based image super-resolution reconstruction system according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a super-resolution image reconstruction method based on multi-level neighborhood embedding according to another embodiment of the present invention;
FIG. 5 shows a block standard and reconstruction diagram for image layers according to an embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
The following describes in detail a specific embodiment of the present invention with reference to fig. 1 to 2.
FIG. 1 is a flowchart illustrating a super-resolution image reconstruction method based on multi-level neighborhood embedding according to an embodiment of the present invention.
FIG. 2 shows a decomposition result diagram of a low resolution image according to one embodiment of the invention.
As shown in FIG. 1, the super-resolution image reconstruction method based on multi-level neighborhood embedding according to one embodiment of the present invention includes: 102, performing controllable pyramid decomposition operation on an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions; 104, respectively reconstructing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions according to a neighborhood embedding algorithm to obtain corresponding high-resolution sub-band images; and 106, performing inverse transformation processing of the controllable pyramid decomposition operation on the plurality of high-resolution sub-band images to generate a high-resolution image.
In the technical scheme, a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, which are obtained by subjecting an image to be processed (such as a single-frame low-resolution image or a low-resolution image sequence) to controllable pyramid decomposition operation, are reconstructed by using a neighborhood embedding algorithm, and then the high-resolution image is generated by inverse transformation of the controllable pyramid decomposition operation, so that the problem that texture details of the image are smoothed due to direct reconstruction of all high-frequency components of the image in the existing image super-resolution reconstruction method is effectively solved, and the accuracy of image super-resolution reconstruction is improved.
Specifically, as shown in fig. 2, the decomposition results of different frequency components are extracted from the input image to be processed by using the self-invertible pyramid transform with the scale of 1 and the direction of 2, which are respectively as follows from left to right: high-pass image, band-pass subband image in both directions (0 °,90 °), low-pass residual image.
In the above technical solution, preferably, the step 102 specifically includes: calculating the response of the image to be processed to a group of multi-directional controllable band-pass filters according to a preset decomposition formula to obtain band-pass sub-band images in multiple directions and the high-pass imageAnd the low-pass residual image, the preset decomposition formula being:
wherein, XtRepresenting the image to be processed in a manner such that,
representing band-pass sub-bands, X, in said plurality of directionst 0Representing said high-pass image, Xt N+1Representing said low-pass residual image, F: (·) And F-1(·) Respectively representing the Fourier transform and the inverse Fourier transform, f (theta)i) Representative direction is θiControllable band-pass filter of giRepresenting a high pass filter or a low pass filter calculated from a controllable band pass filter.
In the technical scheme, a group of multidirectional controllable band-pass filters are used for filtering an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, so that multiple image levels are obtained, and the images of all the image levels are subjected to corresponding neighborhood embedding processing.
In any of the above technical solutions, preferably, the step 104 specifically includes: respectively decomposing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions into corresponding image blocks; searching K adjacent neighborhood image blocks in a dictionary database of a corresponding image layer by utilizing a K-NN algorithm for each image block to obtain K reconstruction coefficients; and obtaining the corresponding high-resolution sub-band image according to the K reconstruction coefficients and the K adjacent domain image blocks.
In the technical scheme, when reconstructing the image of each image level obtained by decomposition according to a neighborhood embedding algorithm, firstly, the corresponding image needs to be decomposed into a plurality of image blocks, and K-NN algorithm (K-Nearest Neighbor, an algorithm for finding Nearest neighbors) is used for each image block to obtain K neighborhood image blocks (i.e. image blocks with high similarity) Nearest to the image block, so as to estimate corresponding K reconstruction coefficients, and then each neighborhood image block is multiplied by the corresponding reconstruction coefficient to combine the multiplication results, so as to obtain a high-resolution subband image with significantly improved resolution, wherein a dictionary database corresponding to the image level refers to all high-frequency image sets, low-frequency image sets and image sets corresponding to each direction formed after decomposing the image to be processed.
To more efficiently estimate more accurate reconstruction coefficients, a neighborhood embedding algorithm is used using l2Norm of
To constrain the least squares problem, it can be expressed as follows:
wherein the image block xt iFor the image block to be reconstructed, Nl iIs xt iThe nearest neighbor image blocks obtained by the K-NN algorithm in the low-resolution dictionary,
for the reconstruction coefficient to be solved, λ is a weight for balancing two terms, and preferably can be 0.15, and then the corresponding reconstructed high-resolution sub-band image block can be represented by
Calculating and integrating the high-resolution sub-band image blocks to obtain a high-resolution sub-band image Yt i。
In any of the above technical solutions, preferably, for the band-pass subband images and the high-pass images in the multiple directions, the K neighboring neighborhood image blocks are searched in a dictionary database of a corresponding image hierarchy according to global feature information and local feature information; and for the low-pass residual image, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to the joint local feature information.
In the technical scheme, the reconstruction quality of each image block depends on the accuracy of the found nearest neighbor image block, so that the standard for selecting similar blocks is crucial, and in order to ensure the reconstruction effect of image blocks of different image levels, for a high-pass image and band-pass sub-band images in multiple directions, the image block characteristic information of the image to be processed is introduced as global characteristic information, and the characteristic information represented by the image blocks of the high-pass image and the band-pass sub-band images in multiple directions is used as local characteristic information; while for low-pass residual images, their gradient features cannot be extracted efficiently because their low frequency is smoother, but they are continuous and already contain sufficient image structure information, thus introducing joint local feature information as a measure of their similarity. In addition, the feature information refers to texture feature information of the image and the like so as to ensure the accuracy of super-resolution reconstruction of the image.
Specifically, for band-pass subband images and high-pass images in multiple directions, image blocks
The distance of the image block to search for the nearest neighbor in K-NN can be defined as follows:
wherein the content of the first and second substances,
represents a gradient operator, xtAnd xs jAre respectively image blocks xt i，xs ijAfter the corresponding image to be processed is interpolated, η is the weight for balancing two terms, preferably 1, of the image block at the same position in the obtained image, where the first term represents the image block xt iAnd xs ijThe second term represents the distance between the two global features.
Whereas for low-pass residual images, image block xt N+1The distance of the image block to search for the nearest neighbor in K-NN can be defined as follows:
in any of the above technical solutions, preferably, the inverse transform processing of the controllable pyramid decomposition operation is performed on a plurality of the high-resolution sub-band images to generate a final high-resolution image YtThis can be represented by the following formula:
wherein, representing high resolution subband images, Y, corresponding to the band pass subbands in the plurality of directionst 0Representing a high-resolution sub-band image, Y, corresponding to said high-pass imaget N+1Representing the high resolution subband image corresponding to said low pass residual image, F: (·) And F-1(·) Respectively representing the Fourier transform and the inverse Fourier transform, f (theta)i) Representative direction is θiControllable band-pass filter of g0Denotes a low-pass filter, g, calculated from a controllable band-pass filterN+1Representing a high pass filter calculated from a controllable band pass filter.
In any of the above technical solutions, preferably, the method further includes: and optimizing the high-resolution image by using a non-local mean algorithm.
In the technical scheme, in order to obtain a better reconstruction effect, a non-local mean algorithm is used for carrying out optimization processing on the high-resolution image obtained through inverse transformation processing of the controllable pyramid decomposition operation so as to reduce reconstruction errors among similar image blocks of the high-resolution image.
In particular for high resolution images YtAny image block y in (1)tFirst, find its similar block y in the whole grapht lAnd limits the minimum reconstruction error, which can be expressed as:
wherein the weight wlDependent on image block yt lAnd ytThen obtaining a final reconstructed high resolution image Y by iterative optimizationt。
FIG. 3 is a block diagram of a super-resolution image reconstruction system based on multi-level neighborhood embedding according to an embodiment of the present invention.
As shown in FIG. 3, the super-resolution image reconstruction system 300 based on multi-level neighborhood embedding according to an embodiment of the present invention comprises: a decomposition module 302, a reconstruction module 304, and a processing module 306.
The decomposition module 302 is configured to perform controllable pyramid decomposition on an image to be processed to obtain a high-pass image, a low-pass residual image, and band-pass sub-band images in multiple directions; a reconstruction module 304, configured to perform reconstruction processing on the band-pass sub-band images in the multiple directions, the high-pass image, and the low-pass residual image, which are obtained by decomposition in the decomposition module 302, according to a neighborhood embedding algorithm, respectively to obtain corresponding high-resolution sub-band images; a processing module 306, configured to perform inverse transformation processing of the controllable pyramid decomposition operation on the multiple high-resolution sub-band images processed by the reconstruction module 304 to generate a high-resolution image.
In the technical scheme, a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, which are obtained by subjecting an image to be processed (such as a single-frame low-resolution image or a low-resolution image sequence) to controllable pyramid decomposition operation, are reconstructed by using a neighborhood embedding algorithm, and then the high-resolution image is generated by inverse transformation of the controllable pyramid decomposition operation, so that the problem that texture details of the image are smoothed due to direct reconstruction of all high-frequency components of the image in the existing image super-resolution reconstruction method is effectively solved, and the accuracy of image super-resolution reconstruction is improved.
In the above technical solution, preferably, the decomposition module 302 is specifically configured to: calculating the response of the image to be processed to a group of multi-directional controllable band-pass filters according to a preset decomposition formula so as to obtain band-pass sub-band images in multiple directions, the high-pass image and the low-pass residual image, wherein the preset decomposition formula is as follows:
wherein，XtRepresenting the image to be processed in a manner such that,
representing band-pass sub-bands, X, in said plurality of directionst 0Representing said high-pass image, Xt N+1Representing said low-pass residual image, F: (·) And F-1(·) Respectively representing the Fourier transform and the inverse Fourier transform, f (theta)i) Representative direction is θiControllable band-pass filter of giRepresenting a high pass filter or a low pass filter calculated from a controllable band pass filter.
In the technical scheme, a group of multidirectional controllable band-pass filters are used for filtering an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions, so that multiple image levels are obtained, and the images of all the image levels are subjected to corresponding neighborhood embedding processing.
In any one of the above technical solutions, preferably, the reconstruction module 304 specifically includes: decomposition submodule 3042, lookup submodule 3044, and processing submodule 3046.
The decomposition sub-module 3042 is configured to decompose the band-pass sub-band images in the multiple directions, the high-pass images, and the low-pass residual images into corresponding image blocks respectively; a searching submodule 3044, configured to search, by using a K-NN algorithm, K neighboring neighborhood image blocks in a dictionary database of a corresponding image hierarchy for each image block obtained by decomposition by the decomposing submodule 3042, so as to obtain K reconstruction coefficients; the processing submodule 3046 is configured to obtain the corresponding high-resolution sub-band image according to the K reconstruction coefficients and the K neighboring domain image blocks obtained by the processing by the searching submodule 3044.
In the technical scheme, when reconstructing the image of each image level obtained by decomposition according to a neighborhood embedding algorithm, firstly, the corresponding image needs to be decomposed into a plurality of image blocks, and K-NN algorithm (K-Nearest Neighbor, an algorithm for finding Nearest neighbors) is used for each image block to obtain K neighborhood image blocks (i.e. image blocks with high similarity) Nearest to the image block, so as to estimate corresponding K reconstruction coefficients, and then each neighborhood image block is multiplied by the corresponding reconstruction coefficient to combine the multiplication results, so as to obtain a high-resolution subband image with significantly improved resolution, wherein a dictionary database corresponding to the image level refers to all high-frequency image sets, low-frequency image sets and image sets corresponding to each direction formed after decomposing the image to be processed.
In any of the above technical solutions, preferably, the search submodule 3044 is specifically configured to: for the band-pass sub-band images and the high-pass images in the multiple directions, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to global characteristic information and local characteristic information; and for the low-pass residual image, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to the joint local feature information.
In the technical scheme, the reconstruction quality of each image block depends on the accuracy of the found nearest neighbor image block, so that the standard for selecting similar blocks is crucial, and in order to ensure the reconstruction effect of image blocks of different image levels, for a high-pass image and band-pass sub-band images in multiple directions, the image block characteristic information of the image to be processed is introduced as global characteristic information, and the characteristic information represented by the image blocks of the high-pass image and the band-pass sub-band images in multiple directions is used as local characteristic information; while for low-pass residual images, their gradient features cannot be extracted efficiently because their low frequency is smoother, but they are continuous and already contain sufficient image structure information, thus introducing joint local feature information as a measure of their similarity. In addition, the feature information refers to texture feature information of the image and the like so as to ensure the accuracy of super-resolution reconstruction of the image.
In any of the above technical solutions, preferably, the method further includes: an optimizing module 308, configured to perform optimization processing on the high-resolution image processed by the processing module 306 by using a non-local mean algorithm.
In the technical scheme, in order to obtain a better reconstruction effect, a non-local mean algorithm is used for carrying out optimization processing on the high-resolution image obtained through inverse transformation processing of the controllable pyramid decomposition operation so as to reduce reconstruction errors among similar image blocks of the high-resolution image.
Another embodiment of the present invention will be described with reference to fig. 4 and 5.
FIG. 4 shows a decomposition result diagram of a low resolution image according to one embodiment of the invention.
FIG. 5 shows a block standard and reconstruction diagram for image layers according to an embodiment of the present invention.
As shown in fig. 4, the method for reconstructing super-resolution image based on multi-level neighborhood embedding according to another embodiment of the present invention specifically includes:
Specifically, fig. 5 shows the image level similarity block standard and reconstruction process for extracting different frequency components from the input low resolution picture and the training set picture by using the self-reversible multidirectional controllable pyramid transform with the scale of 1.
According to the technical scheme, on the basis of a neighborhood embedded framework, the detailed characteristic information such as texture of an image can be reflected in different frequency components, the image characteristics on different frequencies are restored in a targeted mode through image decomposition, the reconstructed image is obtained through inverse transformation of the image decomposition, the high-resolution reconstruction result is superior to the scheme in the related technology in subjective and objective performances, and the high-resolution reconstruction method can be flexibly applied to the fields of interest area amplification and clarification in video monitoring, medical images, satellite images or other high-end multimedia information systems due to the practicability of the high-resolution reconstruction result.
The technical scheme of the invention is described in detail in the above with reference to the attached drawings, and by the technical scheme of the invention, the problem that the texture details of the image are smoothed due to the fact that all high-frequency components of the image are directly reconstructed can be effectively solved, so that the accuracy of super-resolution reconstruction of the image is improved.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. An image super-resolution reconstruction method based on multi-level neighborhood embedding is characterized by comprising the following steps:
carrying out controllable pyramid decomposition operation on an image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions;
reconstructing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions respectively according to a neighborhood embedding algorithm to obtain corresponding high-resolution sub-band images;
and performing inverse transformation processing of the controllable pyramid decomposition operation on the plurality of high-resolution sub-band images to generate a high-resolution image.
2. The image super-resolution reconstruction method based on multi-level neighborhood embedding of claim 1, wherein the performing controllable pyramid decomposition on the image to be processed to obtain a high-pass image, a low-pass residual image and band-pass subband images in multiple directions specifically comprises:
calculating the response of the image to be processed to a group of multi-directional controllable band-pass filters according to a preset decomposition formula so as to obtain band-pass sub-band images in multiple directions, the high-pass image and the low-pass residual image, wherein the preset decomposition formula is as follows:
wherein, XtRepresenting the image to be processed in a manner such that,
representing band-pass sub-bands, X, in said plurality of directionst 0Representing said high-pass image, Xt N+1Representing the low-pass residual image, F (-) and F-1(. cndot.) denotes Fourier transform and inverse Fourier transform, respectively, f (θ)i) Representative direction is θiControllable band-pass filter of giRepresenting a high pass filter or a low pass filter calculated from a controllable band pass filter, N being an integer.
3. The image super-resolution reconstruction method based on multilevel neighborhood embedding of claim 2, wherein reconstructing the band-pass sub-band images, the high-pass images and the low-pass residual images in the plurality of directions according to a neighborhood embedding algorithm to obtain corresponding high-resolution sub-band images comprises:
respectively decomposing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions into corresponding image blocks;
searching K adjacent neighborhood image blocks in a dictionary database of a corresponding image layer by utilizing a K-NN algorithm for each image block to obtain K reconstruction coefficients;
and obtaining the corresponding high-resolution sub-band image according to the K reconstruction coefficients and the K adjacent domain image blocks.
4. The super-resolution image reconstruction method based on multi-level neighborhood embedding of claim 3,
for the band-pass sub-band images and the high-pass images in the multiple directions, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to global characteristic information and local characteristic information;
and for the low-pass residual image, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to the joint local feature information.
5. The super-resolution image reconstruction method based on multi-level neighborhood embedding of any claim 1 to 4, further comprising:
and optimizing the high-resolution image by using a non-local mean algorithm.
6. An image super-resolution reconstruction system based on multi-level neighborhood embedding is characterized by comprising the following steps:
the decomposition module is used for carrying out controllable pyramid decomposition operation on the image to be processed to obtain a high-pass image, a low-pass residual image and band-pass sub-band images in multiple directions;
the reconstruction module is used for respectively reconstructing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions, which are obtained by the decomposition module, according to a neighborhood embedding algorithm to obtain corresponding high-resolution sub-band images;
and the processing module is used for performing inverse transformation processing of the controllable pyramid decomposition operation on the plurality of high-resolution sub-band images obtained by the processing of the reconstruction module so as to generate a high-resolution image.
7. The system of claim 6, wherein the decomposition module is specifically configured to:
calculating the response of the image to be processed to a group of multi-directional controllable band-pass filters according to a preset decomposition formula so as to obtain band-pass sub-band images in multiple directions, the high-pass image and the low-pass residual image, wherein the preset decomposition formula is as follows:
wherein, XtRepresenting the image to be processed in a manner such that,
representing band-pass sub-bands, X, in said plurality of directionst 0Representing said high-pass image, Xt N+1Representing the low-pass residual image, F (-) and F-1(. cndot.) denotes Fourier transform and inverse Fourier transform, respectively, f (θ)i) Representative direction is θiControllable band-pass filter of giRepresenting a high pass filter or a low pass filter calculated from a controllable band pass filter, N being an integer.
8. The system for super-resolution image reconstruction based on multi-level neighborhood embedding of claim 7, wherein the reconstruction module comprises:
the decomposition sub-module is used for decomposing the band-pass sub-band images, the high-pass images and the low-pass residual images in the multiple directions into corresponding image blocks respectively;
the searching submodule is used for searching K adjacent neighborhood image blocks in a dictionary database of a corresponding image layer by utilizing a K-NN algorithm for each image block obtained by decomposing the decomposing submodule so as to obtain K reconstruction coefficients;
and the processing submodule is used for obtaining the corresponding high-resolution sub-band image according to the K reconstruction coefficients and the K adjacent domain image blocks obtained by the processing of the searching submodule.
9. The system of claim 8, wherein the lookup submodule is specifically configured to:
for the band-pass sub-band images and the high-pass images in the multiple directions, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to global characteristic information and local characteristic information;
and for the low-pass residual image, searching the K adjacent neighborhood image blocks in a dictionary database corresponding to the image hierarchy according to the joint local feature information.
10. The system for super-resolution image reconstruction based on multi-level neighborhood embedding of any one of claims 6 to 9, further comprising:
and the optimization module is used for optimizing the high-resolution image obtained by the processing module by using a non-local mean algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610108841.3A CN107133921B (en) | 2016-02-26 | 2016-02-26 | Image super-resolution reconstruction method and system based on multilevel neighborhood embedding |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610108841.3A CN107133921B (en) | 2016-02-26 | 2016-02-26 | Image super-resolution reconstruction method and system based on multilevel neighborhood embedding |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107133921A CN107133921A (en) | 2017-09-05 |
| CN107133921B true CN107133921B (en) | 2020-03-06 |
Family
ID=59720608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610108841.3A Expired - Fee Related CN107133921B (en) | 2016-02-26 | 2016-02-26 | Image super-resolution reconstruction method and system based on multilevel neighborhood embedding |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107133921B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107680037B (en) * | 2017-09-12 | 2020-09-29 | 河南大学 | Improved face super-resolution reconstruction method based on nearest characteristic line manifold learning |
| US11501415B2 (en) | 2019-11-15 | 2022-11-15 | Huawei Technologies Co. Ltd. | Method and system for high-resolution image inpainting |
| CN111935417B (en) * | 2020-07-02 | 2021-12-14 | 清华大学 | Hierarchical video splicing method and device based on multi-scale camera array |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102163329A (en) * | 2011-03-15 | 2011-08-24 | 河海大学常州校区 | Super-resolution reconstruction method of single-width infrared image based on scale analogy |
| EP2662825A1 (en) * | 2012-05-10 | 2013-11-13 | Thomson Licensing | Method and device for generating a super-resolution version of a low resolution input data structure |
| CN103500436A (en) * | 2013-09-17 | 2014-01-08 | 广东威创视讯科技股份有限公司 | Image super-resolution processing method and system |
| CN103593825A (en) * | 2013-10-17 | 2014-02-19 | 西安电子科技大学 | Image super-resolution method based on improved non-local restriction and local self similarity |
| JP2015197818A (en) * | 2014-04-01 | 2015-11-09 | キヤノン株式会社 | Image processing apparatus and method of the same |
| EP2966613A1 (en) * | 2014-07-10 | 2016-01-13 | Thomson Licensing | Method and apparatus for generating a super-resolved image from an input image |
-
2016
- 2016-02-26 CN CN201610108841.3A patent/CN107133921B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102163329A (en) * | 2011-03-15 | 2011-08-24 | 河海大学常州校区 | Super-resolution reconstruction method of single-width infrared image based on scale analogy |
| EP2662825A1 (en) * | 2012-05-10 | 2013-11-13 | Thomson Licensing | Method and device for generating a super-resolution version of a low resolution input data structure |
| CN103500436A (en) * | 2013-09-17 | 2014-01-08 | 广东威创视讯科技股份有限公司 | Image super-resolution processing method and system |
| CN103593825A (en) * | 2013-10-17 | 2014-02-19 | 西安电子科技大学 | Image super-resolution method based on improved non-local restriction and local self similarity |
| JP2015197818A (en) * | 2014-04-01 | 2015-11-09 | キヤノン株式会社 | Image processing apparatus and method of the same |
| EP2966613A1 (en) * | 2014-07-10 | 2016-01-13 | Thomson Licensing | Method and apparatus for generating a super-resolved image from an input image |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107133921A (en) | 2017-09-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhao et al. | Multi-focus image fusion based on the neighbor distance | |
| Jiji et al. | Single‐frame image super‐resolution using learned wavelet coefficients | |
| Saragadam et al. | Miner: Multiscale implicit neural representation | |
| Ma et al. | Sparse representation and position prior based face hallucination upon classified over-complete dictionaries | |
| Dong et al. | Image restoration: a data-driven perspective | |
| CN107133921B (en) | Image super-resolution reconstruction method and system based on multilevel neighborhood embedding | |
| Lei et al. | Remote sensing image denoising based on improved semi-soft threshold | |
| Witwit et al. | Satellite image resolution enhancement using discrete wavelet transform and new edge-directed interpolation | |
| Liu et al. | Research on super-resolution reconstruction of remote sensing images: A comprehensive review | |
| Deeba et al. | Wavelet integrated residual dictionary training for single image super-resolution | |
| Cui et al. | Fast hierarchical deep unfolding network for image compressed sensing | |
| Alvarez-Ramos et al. | Image super-resolution via two coupled dictionaries and sparse representation | |
| Li et al. | Deep scale-aware image smoothing | |
| Karimi et al. | A survey on super-resolution methods for image reconstruction | |
| Shin et al. | Deep orthogonal transform feature for image denoising | |
| Shahdoosti | Two-stage image denoising considering interscale and intrascale dependencies | |
| Ghassab et al. | Plug-and-play video super-resolution using edge-preserving filtering | |
| Aydın et al. | Single-image super-resolution analysis in DCT spectral domain | |
| KR20010074924A (en) | Signal processing | |
| CN111932473A (en) | Phase information noise reduction algorithm of multi-resolution sparse coding and storage medium | |
| Xiao | Research on Super-Resolution Relationship Extraction and Reconstruction Methods for Images Based on Multimodal Graph Convolutional Networks | |
| Abedini et al. | Image Denoising Using Sparse Representation and Principal Component Analysis | |
| Arulmozhi et al. | Detection and diagnosis of pancreatic tumor using convolutional neural network and hybrid particle swarm optimization algorithm (CNN-HPSO) for image classification | |
| Lien et al. | A Novel Image Super-Resolution Technology Based on the Wavelet Coefficients Prediction Scheme | |
| Wang et al. | Super-resolution of compressed images using enhanced attention network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220620 Address after: 100871 No. 5, the Summer Palace Road, Beijing, Haidian District Patentee after: Peking University Patentee after: New founder holdings development Co.,Ltd. Patentee after: BEIJING FOUNDER ELECTRONICS Co.,Ltd. Address before: 100871 No. 5, the Summer Palace Road, Beijing, Haidian District Patentee before: Peking University Patentee before: PEKING UNIVERSITY FOUNDER GROUP Co.,Ltd. Patentee before: BEIJING FOUNDER ELECTRONICS Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20200306 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |