CN111696167A

CN111696167A - Single image super-resolution reconstruction method guided by self-example learning

Info

Publication number: CN111696167A
Application number: CN202010537901.XA
Authority: CN
Inventors: 刘秀萍; 王程
Original assignee: Jingmen Huiyijia Information Technology Co ltd
Current assignee: Jingmen Huiyijia Information Technology Co ltd
Priority date: 2020-06-12
Filing date: 2020-06-12
Publication date: 2020-09-22

Abstract

According to the single-image super-resolution reconstruction method guided by self-paradigm learning, the geometric transformation of the image blocks is introduced, the retrieval space of the internal similar image blocks is expanded, more image blocks with higher similarity can be retrieved by the target image block in the image pyramid, the high-frequency detail information in the reconstructed image obtained by the algorithm is more sufficient, and the reconstruction effect on the internal texture of the image is better; the image internal plane information is obtained by adopting a vanishing point detection method, and the retrieval area is constrained by utilizing the local relevance of the image in the process of retrieving similar image blocks, so that the calculation amount of nearest neighbor search retrieval is greatly reduced, and the algorithm efficiency is improved. Compared with the method in the prior art, the quality of the image reconstructed by the method is greatly improved, the image has higher recovery quality for the internal texture and the edge of the image, and the objective evaluation index of the reconstructed image is also obviously improved.

Description

Single image super-resolution reconstruction method guided by self-example learning

Technical Field

The invention relates to a single-image super-resolution reconstruction method, in particular to a single-image super-resolution reconstruction method guided by self-example learning, and belongs to the technical field of image super-resolution reconstruction.

Background

The resolution of an image is a measure of the amount of information contained in an image, and describes the number of pixels in a unit space. In order to process images more effectively and to make the images more widely and efficiently usable, it is generally desirable to have the quality of the images to be processed as high as possible in practical use. From observation and subjective feelings of users, the high-quality image has richer detail information and is more convenient to distinguish the image edge compared with the low-quality image, pixel points of a unit area are increased by improving the resolution of the image, the description of the internal details of the image is richer, and the aim of fully improving the image quality is achieved. In the actual imaging process of the image, because the image is degraded due to the blur and the motion blur generated by environmental factors, on the other hand, in the transmission and storage process of the image, the loss of the information acquired by the image information transmission end can be caused due to bandwidth limitation, file compression and the like, and the finally acquired image and the observation target have larger degradation, the image is defined as the low-resolution image, and the low-resolution image has a series of adverse effects on the later-stage processing of the image.

The method is difficult to realize technical innovation in a short period, and high-precision imaging equipment is very expensive, so that although the method for improving the image quality through hardware has effective effects, the method is limited by the fact that the technology and the cost are difficult to realize in a short period and almost has no application value; and the other method is to adopt a super-resolution image reconstruction technology, recover a high-resolution image in the same scene through one or more low-resolution images by using a digital image processing method, and reduce the influence of hardware condition limitation on image imaging quality, so that the technology has important development prospect and practical application value.

The super-resolution image reconstruction enables important research and development and application problems in the field of computer vision to improve the effectiveness of images in subsequent application through restoration of high-frequency information of the images, and the development of the super-resolution image reconstruction will be applied to other fields of computer vision such as: positive influence is brought to problems such as pattern recognition and image understanding, and super-resolution image reconstruction has wide application: firstly, in the field of safety monitoring, super-resolution image reconstruction is carried out on an image acquired by monitoring acquisition equipment, a low-cost safety monitoring system can also acquire a relatively good imaging effect, more effective information can be acquired, and a target can be detected more accurately; in the field of remote sensing imaging, the satellite image acquisition equipment is limited by volume and weight and is difficult to update and maintain, so that an observation image with the best quality cannot be obtained. The super-resolution image reconstruction can effectively improve the quality of a target image, so that the application effect of the remote sensing image in the fields of natural environment monitoring, disaster forecast evaluation, remote sensing, aviation exploration and the like is greatly improved; thirdly, in the field of medical imaging, a medical image with higher quality is obtained through super-resolution image reconstruction, and doctors are assisted to judge affected parts and illness states of patients more accurately; and fourthly, in the field of electronic consumption, due to the limitation of image acquisition equipment and transmission, an image with lower resolution is obtained, and the super-resolution image is utilized to reconstruct an image with higher output resolution, so that the visual effect of the image is improved.

The super-resolution image reconstruction adopts a signal processing technology to reconstruct a high-resolution image of the same scene through one or more low-resolution images to be reconstructed. Super-resolution image reconstruction was proposed for the first time in 1964, and an interpolation method for improving the image spatial resolution by linear or spline interpolation was proposed, but the effect of introducing the super-resolution image reconstruction into practice is not ideal. And then, a high-resolution image is reconstructed by a plurality of undersampled sequence images, and the sequence images provide differentiated displacement information which can be complemented, so that a single image with higher quality can be reconstructed. The field is widely concerned later, and at the end of the 20 th century, the super-resolution image reconstruction technology gradually becomes a hotspot in the international image restoration field, along with the development of the fields of mathematical modeling, optimization theory, machine learning and the like.

The reconstruction of the super-resolution images of the images is divided into three types according to different input image forms, namely, reconstruction of the super-resolution images of the sequence images, reconstruction of single-image super-resolution images and reconstruction of the super-resolution images of multi-view images, wherein the reconstruction technology of the single-image super-resolution images is the basis of the reconstruction of the other two types of images. The super-resolution reconstruction method for a single image can be divided into the following steps: the method comprises a single-image interpolation method, a single-image reconstruction method and a single-image example learning method.

The method is a single-image interpolation method, and the super-resolution image reconstruction method based on interpolation is used for interpolating missing high-resolution pixels through a certain interpolation function on the premise that the local structure of an image is smooth without changing the pixels of a low-resolution image. The interpolation algorithm in the prior art is polynomial interpolation, and the selected interpolation basis functions can be classified into nearest neighbor interpolation, bicubic interpolation and the like. The interpolation method utilizes the pixel value of the known pixel point to estimate the unknown point, the internal information of the image is not considered, the image quality is still low although the resolution is improved, and the images output after interpolation have ringing effect, aliasing effect, blocking effect, fuzzy effect and the like.

And secondly, a single image reconstruction method, which combines a low-resolution image and an observation model to provide constraint for image super-resolution image reconstruction by modeling the degradation process of the image based on the reconstruction method. Such algorithms include iterative backprojection and probability analysis. The solution of the iterative back projection method is related to and not unique to the initial value, the selection of a back projection operator is difficult, and the local part of the reconstructed image can generate a sawtooth effect. The probability analysis method also models the degradation process of the image and solves the super-resolution problem by adopting a statistical estimation optimization thought. The super-resolution algorithm based on probability analysis takes the prior knowledge of the high-resolution image as a constraint term of an optimization problem in a prior probability density function mode, so that the solving accuracy is improved.

Third, a single image example learning method, the prior art proposes to use a training data set to obtain prior knowledge of mapping relationship of high and low resolution images to guide image reconstruction, and to use a markov network model to learn the mapping relationship between high and low resolution image blocks for a training set of high and low resolution images, thereby estimating high frequency information of the input low resolution images. The defined nearest neighbor does not consider the geometric deformation of the image block, and the similarity comparison is carried out on the image block only, so that the matching deviation of the method is large, and the similarity is low.

In summary, the present invention is intended to solve the following problems, in view of some of the drawbacks of the prior art:

firstly, the mode of improving the image resolution in the prior art mainly obtains an image with higher resolution by improving the hardware performance of imaging equipment, a sensor for image acquisition mainly comprises a CMOS (complementary metal oxide semiconductor) and a CCD (charge coupled device), the purpose of improving the resolution can be achieved by improving the size of a photosensitive component or reducing the size of pixel points, but the energy consumption is higher due to the improvement of the size of the photosensitive component, more and more noises are introduced by excessively reducing the size of the pixel under the existing technical condition, the method is difficult to realize technical innovation in a short period, and the high-precision imaging equipment is very expensive, so the mode of improving the image quality through hardware has effective effects but is limited by the fact that the technology and the cost are difficult to realize in a short period, and almost has no application value; and by adopting a super-resolution image reconstruction technology and utilizing a digital image processing method, the influence of hardware condition limitation on the image imaging quality is reduced, and the method has important development prospect and practical application value.

Secondly, in the super-resolution image reconstruction method based on interpolation in the prior art, on the premise of not changing the pixels of the low-resolution image, the local structure of the image is assumed to be smooth, and the missing high-resolution pixels are interpolated through a certain interpolation function. The interpolation method utilizes the pixel value of the known pixel point to estimate the unknown point, the internal information of the image is not considered, the image quality is still low although the resolution is improved, and the images output after interpolation have ringing effect, aliasing effect, blocking effect, fuzzy effect and the like. The method has great limitation, most scenes cannot be suitable, and the method has little practical value.

Third, most of the prior art methods based on self-paradigm learning use self-approximating features in multiple scales and image pyramids to represent similar image blocks searched in the downsampled image. However, since the original image has a lot of complicated texture features inside, the self-example cannot reconstruct all details of the image well due to the quantity limitation. The loss of image detail information is serious, the searching space of the internal image block is small, the matching of multi-scale similar image blocks is large, and the similarity between the matched image blocks is low.

Fourthly, the PatchMatch algorithm in the prior art adopts random or prior information to initialize a nearest neighbor search space, and for a solution space of 7-dimensional geometric transformation of an image block, the convergence efficiency of a random initialization method is low, so that the situation of local optimal solution cannot be avoided.

Disclosure of Invention

Aiming at the defects of the prior art, the single-image super-resolution reconstruction method guided by the self-paradigm learning provided by the invention expands the retrieval space of the internal similar image blocks by introducing the geometric transformation of the image blocks, so that more image blocks with higher similarity can be retrieved by the target image block in the image pyramid, the high-frequency detail information in the reconstructed image obtained by the algorithm of the invention is more sufficient, and the reconstruction effect on the internal texture of the image is better; the image internal plane information is obtained by adopting a vanishing point detection method, and the retrieval area is constrained by utilizing the local relevance of the image in the process of retrieving similar image blocks, so that the calculation amount of nearest neighbor search retrieval is greatly reduced, and the algorithm efficiency is improved. Compared with the method for learning sparse representation by self-paradigm in the prior art, the quality of the image reconstructed by the method is greatly improved, the image has higher recovery quality for the internal texture and the edge of the image, and the objective evaluation index of the reconstructed image is also obviously improved

In order to achieve the technical effects, the technical scheme adopted by the invention is as follows:

according to the super-resolution reconstruction method of the single image guided by the example learning, geometric invariance of an image block is introduced into similar image block matching, a single-image super-resolution image reconstruction algorithm based on the example geometric invariance is realized, and in the similar image block matching process, the geometric deformation of the image block is used as a loss function to expand a search space of an internal image block;

the single-image super-resolution reconstruction model guided by the example learning is divided into three parts, namely: defining nearest neighbor structure, searching nearest neighbor, and synthesizing high-resolution image block; the method is characterized in that a single-image super-resolution reconstruction model guided by example learning is used for super-resolution image reconstruction, and the whole method flow comprises the following three steps: respectively as follows: image detection and pyramid construction, similar image block matching and multi-stage amplified image reconstruction;

the whole method flow of the single image super-resolution reconstruction method guided by the self-example learning comprises the following steps:

the first step, image detection and pyramid construction,

in the image detection and pyramid construction stage, image internal plane information is obtained based on vanishing point detection, and an image pyramid is established, wherein the image pyramid comprises three parts: image plane detection, acquisition and utilization of transmission transformation matrix, image pyramid structure,

1. image plane detection: vanishing point detection is carried out on the input low-resolution image, and the vanishing point and the corresponding vanishing line of the image are obtained

Gathering and then passing the vanishing line

The density distribution maps are paired pairwise to obtain plane information;

2. transmission transformation matrix B_CObtaining utilization of vanishing line

Obtaining a transmission transformation matrix B of the corresponding plane_CPerforming geometric correction to ensure that parallel lines in the three-dimensional space are still parallel in the image;

3. image pyramid construction: firstly, an input low-resolution image is subjected to continuous down-sampling to construct a self full-frequency image pyramid, then bicubic interpolation is carried out on each layer of image of the full-frequency image pyramid to construct a low-frequency image pyramid, a high-low resolution image pair is constructed, and a full-frequency image corresponding to the high-low resolution image pyramid is reconstructed through a low-frequency image of a target scale;

in the second step, similar image block matching is performed,

1. image blocking: partitioning the low-resolution image with the target scale lacking high-frequency information in the low-frequency image pyramid, partitioning the low-resolution image with the size of i x i, and overlapping adjacent image blocks to obtain a large number of low-resolution image blocks;

2. similar image block matching: adopting an image block geometric transformation model in the low-frequency image pyramid to perform similar image block matching on a target image block;

thirdly, the image is amplified in multiple stages and reconstructed,

1. applying the mapping relation learned in the low-resolution image pyramid to the high-resolution image pyramid to obtain a high-resolution image with a target scale; the transformation relation T between the target block and the similar block can be obtained in the low-resolution image pyramid, and the high-resolution image block can be obtained by applying the inverse matrix of T to the high-resolution image block at the same position;

2. adopting a multi-stage amplified super-resolution image reconstruction method, and carrying out error correction on the high-resolution image obtained by reconstructing the high-resolution image by an iterative back projection method;

b obtained after error correction_i+1As input image, repeating the iteration until B_i+CWhen the target resolution is reached, outputting a super-resolution image reconstruction result;

through the three processes, the low-resolution image is input and reconstructed into the high-resolution image under the condition of not depending on an external training data set.

The invention relates to a single-image super-resolution reconstruction method guided by example learning, in particular to a vanishing point detection method for extracting plane information, which comprises the following steps of:

firstly, carrying out edge detection on an image to be detected, and filling line segments into a known area of the image;

step two, acquiring at most 3 vanishing points by adopting the vanishing lines detected in the step one and adopting a random sampling consistency algorithm;

step three, pairwise matching is carried out by utilizing at most 3 vanishing points obtained by clustering to obtain 3 planes;

using vanishing lines

Parameters representing the plane C, by a transmission transformation matrix B_CPerforming geometric correction on the transmission image of the real plane to ensure that line segments parallel to each other in the three-dimensional space are still parallel in the imaging image;

a plane in an image is determined by parallel lines in two groups of three-dimensional spaces, two groups of vanishing lines belonging to different vanishing points can exist in the same image plane area, the vanishing point plane detection method determines a corresponding plane area by positioning the overlapping area of the two groups of vanishing lines belonging to different vanishing points, and the detection process comprises the following steps:

step 1, carrying out Gaussian kernel diffusion on evanescent lines of each group of evanescent points in an image to obtain a density distribution map of the corresponding evanescent points in the image;

and 2, multiplying the vanishing point corresponding to the vanishing line density distribution map pairwise to obtain a planar density map of the corresponding plane, wherein the output picture can well reflect the overlapped part of the two line segments.

The single-image super-resolution reconstruction method based on the example learning guidance further comprises three parts, namely a coplanar loss function, a line segment direction loss function and a neighbor searching loss function;

coplanar loss function: the image blocks in the same plane have local approximate characteristics, a loss function of plane compatibility is added, and the association degree of the target image block and the search space is higher;

line segment directional loss function: combining the characteristic that the texture features of the artificial scene tend to reappear in the vertical and horizontal directions, adding a loss function related to the direction;

neighbor search loss function: the search space is restricted to the local area to improve the image reconstruction quality.

The super-resolution reconstruction method of single image guided by self-example learning further comprises the step of utilizing an image internal plane vanishing line obtained by detecting and positioning an image internal plane in the definition of the nearest neighborhood structure

Deriving the transmission variation of the corresponding planeChange matrix

Carrying out geometric transformation on the image block in the nearest neighborhood searching space to expand the searching space;

assume that a target image block inside each low resolution image I to be reconstructed is denoted as M (t)_i) Wherein t is_iAs the central coordinates of the target image block

The geometric transformation between the target image block and the best similar image block is defined as T_iIn simple form, the following:

M(t_i)≈T_i*M(S_i)

wherein M (S)_i) Representing the best similar image block matched by the nearest neighbor search retrieval algorithm,

representing its center coordinates;

the geometric transformation of the image block is unified into transmission transformation and affine transformation from an image geometric transformation model, and the acquisition of the image internal plane information adopts a vanishing point detection method and passes through a vanishing line

Obtaining a transmission transformation relation of a corresponding plane, independently converting affine transformation and transmission transformation into two transformation matrixes, and independently solving;

hypothetical geometric transformation T in the process of image-like block matching_i(b_i) The matching image block is M (S)_i) Suppose N (t)_i,b_i) For geometrically transformed similar image blocks:

N(t_i,b_i)＝T_i(b_i)M(S_i)。

from the super-resolution reconstruction method of the single image guided by example learning, further, the similarity loss function adopts the Euclidean distance square sum weighted by RGB space Gaussian to measure the image block M (t) of the target block_i) Similar image block N (t) after geometric transformation_i,b_i) Loss function D of similarity between_similarity(t_i,b_i) The form is as follows:

G_ifor Gaussian weights, search for similar image blocks N (t)_i,b_i) Is transformed into a defined geometric transformation T_i(b_i) From the image geometric transformation model, the invention is directed to geometric transformation T_i(b_i) The definition is divided into two parts, respectively affine transformation T_a(b) And transmission transformation

Affine transformation T of image blocks_a(b) Is formed by combining rotation transformation, scale transformation and shear transformation, and has the following form:

wherein, the matrix

The scale is represented by a transformation of the scale,

the parameters are scaled as follows:

wherein, the matrix

Representing the rotation transformation, b is the rotation transformation angle parameter form as follows:

wherein the miscut transform matrix

The form is as follows:

in summary, in combination with transmission transformation

Definition of (1), global geometric transformation T_i(b_i) Is represented as follows:

wherein the parameter b in the geometric transformation_i＝(s_i，C_i)，

Representing 7 dimensional geometric transformation freedom, C, of similar image blocks to be matched_iA plane parameter representing vanishing point detection acquisition;

the invention provides a geometric transformation model T_i(b_i) Decomposing the transmission deformation matrix into four independent geometric transformation matrixes which are connected in series and respectively comprise: transmission transformation matrix, similarity matrix, miscut matrix, affine matrix, independent geometric transformation model pass pair

Estimation of the parameters determines S, E and F matrices in the affine transformation, and utilizes the target image block M (t)_i) Similar image block M (S) to the best match_i) Coordinate to transmission transformation of

Carrying out accurate estimation, combining the characteristics of segmentation and smoothing of the original image, and geometrically transforming the model T_i(b_i) The efficiency of nearest neighbor estimation is improved.

Further, an image block scale loss function is added in the image block matching process:

D_measure＝g_measuremin(0，SRF-measure(T_i))

where SRF denotes the super-resolution target scale, g_measureMeasure (T) as a parameter of the size loss function_i) Representing geometric transformations T_iThe scale information of (a) is in the form of:

wherein T is_1,1Representing a geometric transformation matrix T_iIn combination with the planar loss function D_plane(t_i，b_i) The final given objective function for the nearest neighbor in retrieving similar image blocks is defined as follows:

wherein Ω is the low resolution image block M (t)_i) Set of pixels in (b)_iThe geometric transformation parameters of the corresponding image block are included.

The method for reconstructing the super-resolution of the single image guided by the example learning further adopts a PatchMatch algorithm in the similar image block matching process, and redefines a search strategy of the PatchMatch algorithm according to the algorithm characteristics of the invention;

the method specifically redefines the processes of nearest neighbor matching initialization, neighborhood forward and reverse propagation and random error item searching in the PatchMatch algorithm.

From the example learning-guided single-image super-resolution reconstruction method, further, neighborhood forward and backward propagation is based on the relevance of the local structure of the image, that is, the target image block P and the best matching image block Q are in a similar image block relationship, then the best matching image block of the image block adjacent to P tends to be located in the adjacent area of Q, and the propagation process of approximate nearest neighbor search in the patch match algorithm includes: forward propagation and backward propagation;

when the iteration number is even, the forward propagation is carried out, and the target image block M (t)_i) Using the best matching image block corresponding to the left (x-1, y) and the upper (x, y-1) to carry out similarity comparison, and updating the geometric transformation parameter b according to the result_i＝(s_i,c_i) The matching accuracy is improved;

and when the iteration times are odd, performing back propagation, considering the optimal matching image blocks corresponding to the right (x +1, y) and the lower (x, y +1) of the target image block, and improving M (t)_i) The matching accuracy of (2).

From a single image super-resolution reconstruction method guided by example learning, further, the synthesis of a high-resolution image block adopts a nearest neighbor search retrieval method to obtain a target image block M in a low-frequency image pyramid_A(t_i) Approximate nearest neighbor N of_A(S_i) And geometric transformation T_i(b_i) Extracting corresponding high-resolution version N in the high-resolution image pyramid_B(S_i) Then, the corresponding formula of the high resolution image block in the target image is as follows:

M_B(t_i)＝N_B(S_i)T_i ^-1(b_i)

in the reconstruction process, the super-resolution image reconstruction is carried out by adopting multi-stage amplification, and the target resolution image J is not directly obtained_BFirst obtaining J by zooming in step by step_A+1，J_A+1Inputting a low-resolution upper-level image, and then inputting J_A+1As an input image pair J_A+2Performing reconstruction, and when the resolution improvement scale is 2, making B equal to A + 3;

between each stage of amplification, the iterative back projection method is adopted to carry out degradation processing on the currently output high-resolution image and input the low-resolution image J_BError comparison is carried out, the error is back projected to the current reconstructed image for error correction, and each level of reconstructed image and the input low-resolution image J are ensured_BMaintaining consistent characteristics。

Compared with the prior art, the invention has the advantages and innovation points that:

according to the single-image super-resolution reconstruction method guided by self-paradigm learning, the inspiration that image detail information can cause deformation of an object on an imaging plane due to bending of the plane and transmission transformation of camera imaging in most natural or artificial scenes is adopted, and on the basis that the searching space of internal similar image blocks is expanded by constructing a multi-scale image pyramid in self-paradigm learning in the prior art, geometric invariance of image blocks is introduced in the similar image block searching process, the searching space of the internal image blocks is expanded, and the similarity between matched image blocks is higher while the matching deviation of the multi-scale similar image blocks is reduced.

The invention introduces the geometric invariance of the image block into the matching of similar image blocks by combining the super-resolution image reconstruction characteristics of self-example learning, realizes the single-image super-resolution image reconstruction algorithm based on the geometric invariance of the example, enlarges the search space of the internal image block by taking the geometric deformation of the image block as a loss function in the matching process of the similar image blocks, achieves the purposes of reducing matching deviation and improving the similarity of the image block, and simultaneously provides more high-frequency information for the reconstruction process.

Thirdly, the method for reconstructing the super-resolution of the single image guided by the self-paradigm learning provided by the invention is used for improving the search strategy of PatchMatch two points aiming at the problem of complex matching solution space of similar images caused by introducing the geometric transformation of the image block so as to improve the matching efficiency of the similar image block in the algorithm of the invention. The method comprises the steps of firstly searching near a target image block by using local similarity of images in an initialization stage to improve the convergence efficiency of an algorithm, and secondly constraining in a random error item searching stage by using plane distribution information obtained by vanishing point detection to improve the correlation between the image blocks.

Fourthly, geometric invariance of image blocks is introduced into matching of similar image blocks, a single-image super-resolution image reconstruction algorithm based on the example geometric invariance is realized, multiple types of test images are tested, and results are evaluated by using a main-stream image evaluation method.

Fifthly, the single-image super-resolution reconstruction method guided by the self-paradigm learning provided by the invention expands the retrieval space of the internal similar image blocks by introducing the geometric transformation of the image blocks, so that more image blocks with higher similarity can be retrieved from the target image block in the image pyramid, the high-frequency detail information in the reconstructed image obtained by the algorithm of the invention is more sufficient, and the reconstruction effect on the internal texture of the image is better; the image internal plane information is obtained by adopting a vanishing point detection method, and the retrieval area is constrained by utilizing the local relevance of the image in the process of retrieving similar image blocks, so that the calculation amount of nearest neighbor search retrieval is greatly reduced, and the algorithm efficiency is improved.

Drawings

FIG. 1 is a schematic diagram of a super-resolution image synthesis process according to the present invention.

FIG. 2 is a flow chart of a single image super-resolution reconstruction method guided by self-example learning according to the present invention.

FIG. 3 is a comparison graph of super-resolution reconstruction effect of single image under different methods of the present invention.

Detailed Description

The following describes a technical solution of the super-resolution reconstruction method for single image guided by example learning with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention and can implement the same.

The self-example and input images are high in relevance, the super-resolution image reconstruction method based on self-example learning has a good restoration effect on texture features in the images, and compared with other learning-based methods, the method is better in adaptability to different super-resolution multiples and to-be-reconstructed images. Because of the numerous and complicated texture features in the original image, the method cannot better reconstruct all the details of the image due to the limitation of the number of the examples, and the problem to be solved by the invention is to expand the search space of similar image blocks in the image.

The invention provides a single-image super-resolution reconstruction method guided by self-example learning, which introduces the geometric invariance of an image block into similar image block matching to realize a single-image super-resolution image reconstruction algorithm based on the example geometric invariance, and in the similar image block matching process, the geometric deformation of the image block is used as a loss function to expand the search space of an internal image block;

the single-image super-resolution reconstruction model guided by the example learning is divided into three parts, namely: defining nearest neighbor structure, searching nearest neighbor, and synthesizing high-resolution image block; the method is characterized in that a single-image super-resolution reconstruction model guided by example learning is used for super-resolution image reconstruction, and the whole method flow comprises the following three steps: image detection and pyramid construction, similar image block matching and multi-stage amplified image reconstruction.

Single-image super-resolution reconstruction algorithm guided by self-example learning

The present invention uses a super-resolution image reconstruction model according to example geometric invariance for super-resolution image reconstruction. The whole process flow of the method comprises three steps: image detection and pyramid construction, similar image block matching and multi-stage amplified image reconstruction.

Image detection and pyramid construction

In the image detection and pyramid construction stage, image internal plane information is obtained based on vanishing point detection, and an image pyramid is established, wherein the image pyramid comprises three parts: image plane detection, transmission transformation matrix acquisition and utilization and image pyramid construction.

Gathering and then passing the vanishing line

And pairing the density distribution maps to obtain plane information.

Obtaining a transmission transformation matrix B of the corresponding plane_CGeometric correction is performed so that the parallel lines in three-dimensional space remain parallel in the image.

3. Image pyramid construction: firstly, an input low-resolution image is subjected to continuous down-sampling to construct a self full-frequency image pyramid, then bicubic interpolation is carried out on each layer of image of the full-frequency image pyramid to construct a low-frequency image pyramid, a high-low resolution image pair is constructed, and a full-frequency image corresponding to the high-low resolution image pyramid is reconstructed through a low-frequency image of a target scale.

(II) similar image block matching

1. Image blocking: and (3) carrying out blocking processing on the low-resolution image with the target scale lacking high-frequency information in the low-frequency image pyramid, blocking the low-resolution image with the size of i x i, and overlapping adjacent image blocks to obtain a large number of low-resolution image blocks.

2. Similar image block matching: and adopting an image block geometric transformation model in the low-frequency image pyramid to perform similar image block matching on the target image block.

(III) Multi-stage magnified image reconstruction

1. And applying the mapping relation learned in the low-resolution image pyramid to the high-resolution image pyramid to obtain the high-resolution image with the target scale. The transformation relation T between the target block and the similar block can be obtained in the low-resolution image pyramid, and the high-resolution image block can be obtained by applying the inverse matrix of T to the high-resolution image block at the same position.

2. The algorithm adopts a multi-stage amplification super-resolution image reconstruction method, and error correction is carried out on the high-resolution image obtained by reconstruction of the obtained high-resolution image through an iterative back projection method.

B obtained after error correction_i+1As input image, repeating the iteration until B_i+CAnd outputting a super-resolution image reconstruction result when the target resolution is reached.

A flow chart of a single-image super-resolution reconstruction method guided by example learning is shown in fig. 2.

In order to obtain a complete high-resolution image, all reconstructed output high-resolution image blocks are spliced. In order to obtain better visual effect, certain overlapping parts exist among input image blocks, correspondingly, the overlapping parts exist when each high-resolution image block which is reconstructed and output is combined, and the pixel average value is adopted for processing the overlapping parts, so that the local smoothness and the structural characteristics of the reconstructed image are ensured.

Second, image internal plane detection and positioning

The plane information extraction adopts a vanishing point detection method, firstly, plane information in a two-dimensional image is extracted through image edge detection and vanishing point clustering, and the method specifically comprises the following steps:

using vanishing lines

Parameters representing the plane C, by a transmission transformation matrix B_CAnd performing geometric correction on the transmission image of the real plane, so that the line segments parallel to each other in the three-dimensional space are still parallel in the imaging image.

step 1, carrying out Gaussian nuclear diffusion on evanescent lines of each group of evanescent points in the image to obtain a density distribution diagram of the corresponding evanescent points in the image.

The plane guide comprises three parts, namely a coplanar loss function, a line segment direction loss function and a neighbor searching loss function

Coplanar loss function: because the image blocks in the same plane have local approximate characteristics, a loss function of plane compatibility is added to make the correlation degree between the target image block and the search space higher.

Line segment directional loss function: in combination with the feature that the texture features of the artificial scene tend to reproduce in both vertical and horizontal directions, the present invention adds a loss function with respect to direction.

Neighbor search loss function: the image reconstruction quality can be improved by restricting the search space in a local area, and the close neighbor search loss function restriction avoids copying image blocks from extremely different scales.

Three, single image super-resolution reconstruction model guided by self-example learning

The single image super-resolution reconstruction guided by the example learning is divided into three parts, namely: defining nearest neighbor structure, searching nearest neighbor, and synthesizing high-resolution image block.

The method has the advantages that a large number of complicated texture features are arranged in the original image, the self-paradigm is limited by the number and can not better reconstruct all details of the image.

(ii) nearest neighbor construction definition

The method utilizes the image internal plane to detect, position and obtain the image internal plane vanishing line

Deriving a transmission transformation matrix for the corresponding plane

And performing geometric transformation on the image blocks in the nearest neighbor search space, so as to expand the search space and improve the matching accuracy of similar image blocks.

For the target image block M (t) in practical application_i) The size is typically selected to be 3 x 3 or 5 x 5, and the reproduction of similar image blocks in the multi-scale image pyramid is more common.

M(t_i)≈T_i*M(S_i)

representing its center coordinates.

And acquiring a transmission transformation relation of the corresponding plane, and independently converting the affine transformation and the transmission transformation into two transformation matrixes to be solved independently.

N(t_i,b_i)＝T_i(b_i)M(S_i)。

(1) similarity loss function

The invention measures the image block M (t) of the target block by adopting the Euclidean distance square sum weighted by RGB space Gaussian_i) Similar image block N (t) after geometric transformation_i,b_i) Loss function D of similarity between_similarity(t_i,b_i) The form is as follows:

wherein, the matrix

The scale is represented by a transformation of the scale,

the parameters are scaled as follows:

wherein, the matrix

wherein the miscut transform matrix

The form is as follows:

in summary, in combination with transmission transformation

wherein the parameter b in the geometric transformation_i＝(s_i，c_i)，

Representing 7 dimensional geometric transformation freedom, C, of similar image blocks to be matched_iRepresenting the plane parameters acquired by the vanishing point detection.

The invention provides a geometric transformation model T_i(b_i) Decomposing a transmission deformation matrix intoThe four independent geometric transformation matrixes are connected in series and respectively comprise: transmission transformation matrix, similarity matrix, miscut matrix, affine matrix, independent geometric transformation model

Accurately estimating, combining the characteristics of the original image segmentation smoothing, and obtaining the geometric transformation model T_i(b_i) The efficiency of the nearest neighbor estimation will be improved.

(2) Scale loss function

Introducing continuous geometric transformations between the target image block and the retrieved nearest neighbor image block may trap the self-paradigm-based super-resolution problem into a locally optimal solution, such as image block M (S) in low-resolution image J_i) And downsampling the image J_fMiddle self M_d(t_i) Most similarly, this makes the super-resolution based on example learning become a bicubic interpolation algorithm. In order to avoid the situation of falling into the local optimal solution, an image block scale loss function is added in the image block matching process:

D_measure＝g_measuremin(0,SRF-measure(T_i))

wherein T is_1,1Representing a geometric transformation matrix T_iThe first row and the first column of (c) encourages the matching image block to include more information while searching for similar image blocks, providing more high frequency information in the super-resolution image reconstruction.

Combined plane loss function D_plane(t_i，b_i) The final given objective function for the nearest neighbor in retrieving similar image blocks is defined as follows:

(II) nearest neighbor search

The PatchMatch algorithm is adopted in the similar image block matching process, and the search strategy of the PatchMatch algorithm is redefined according to the algorithm characteristic of the invention, so that the algorithm of the invention is better adapted.

The invention uses similarity loss function D between image blocks_similarity(t_i，b_i) Affine transformation and plane transmission transformation of the image blocks are introduced, and Euclidean distance comparison is simply carried out on the target image block and the image block to be matched by a super-resolution method based on self-example learning, and the target image block and the image block to be matched are converted into 7-dimensional geometric transformation images of the target image block and the image block to be matched to carry out similarity measurement. However, the more complicated solution space reduces the efficiency of the PatchMatch algorithm based on the two-dimensional image block search method.

b_i＝(s_i,C_i)

Aiming at the characteristic of high complexity of a super-resolution algorithm based on self-example geometric invariance, the method redefines the processes of nearest neighbor matching initialization, neighborhood forward and reverse propagation and random error item searching in the PatchMatch algorithm, and improves the retrieval efficiency of similar image block matching.

(1) Nearest neighbor matching initialization

In the PatchMatch algorithm, a nearest neighbor search space is initialized by adopting random or prior information, the retrieval of similar image blocks is high in efficiency, but the convergence efficiency of a random initialization method is low for a solution space of 7-dimensional geometric transformation of the image blocks.

The invention adopts nearest neighbor matching initialization, and the initialization distance of the nearest neighbor searching space is an area with the distance from a target image block as 0 and the size as the super-resolution magnification factor. And the third step of random error term searching is matched, so that the convergence efficiency of the algorithm is improved on one hand, and the situation of local optimal solution is avoided on the other hand.

(2) Neighborhood forward and backward propagation

The neighborhood forward and backward propagation is based on the relevance of the image local structure, that is, the target image block P and the best matching image block Q are in a similar image block relationship, the best matching image block of the image block adjacent to P tends to be located in the adjacent area of Q, and the propagation process of the approximate nearest neighbor search in the patch match algorithm includes: forward propagation and backward propagation.

When the iteration number is even, the forward propagation is carried out, and the target image block M (t)_i) Using the best matching image block corresponding to the left (x-1, y) and the upper (x, y-1) to carry out similarity comparison, and updating the geometric transformation parameter b according to the result_i＝(s_i,c_i) And the matching accuracy is improved.

(3) Random error term search

Compared with the PatchMatch algorithm which is originally similar and different in comparison between the globally randomly selected image block and the optimally matched image block, the method utilizes the image internal plane parameters obtained by detecting and positioning the image internal plane, adopts the plane prior information to constrain the random area, enables the randomly selected image block and the target image block to be positioned on the same plane, and can improve the efficiency of the algorithm while avoiding the matching deviation through redefined random disturbance search.

(III) Synthesis of high resolution image blocks

By usingThe nearest neighbor searching and searching method of the invention obtains the target image block M in the low-frequency image pyramid_A(t_i) Approximate nearest neighbor N of_A(S_i) And geometric transformation T_i(b_i)。

The super-resolution image synthesis process is shown in fig. 1.

Extracting N of corresponding high-resolution version in high-resolution image pyramid_B(S_i) Then, the corresponding formula of the high resolution image block in the target image is as follows:

M_B(t_i)＝N_B(S_i)T_i ^-1(b_i)

in the reconstruction process, the invention adopts multi-stage amplification to reconstruct the super-resolution image without directly obtaining the target resolution image J_BFirst obtaining J by zooming in step by step_A+1，J_A+1Inputting a low-resolution upper-level image, and then inputting J_A+1As an input image pair J_A+2The reconstruction is performed, and when the resolution enhancement scale is 2, B is made equal to a + 3.

In each stage of amplification, the invention adopts an iterative back projection method to carry out degradation processing on the currently output high-resolution image and input the low-resolution image J_BError comparison is carried out, the error is back projected to the current reconstructed image for error correction, and each level of reconstructed image and the input low-resolution image J are ensured_BThe consistent characteristics are maintained.

Fourth, experimental comparative analysis

The experimental part compares the single-image super-resolution reconstruction method guided by the self-example learning provided by the invention with some mainstream super-resolution algorithms in the prior art, selects images of people and landscape buildings from a super-resolution image library for comparison test, and verifies the reconstruction performance of the method provided by the invention on various image super-resolution images.

In the experiment, the test image is subjected to image reconstruction with the super-resolution multiple of 2 by four super-resolution methods, namely a bicubic interpolation method, an SCSR method, a Self-example learning guided single-image super-resolution reconstruction method and the test image super-resolution reconstruction method, in sequence, and the result is analyzed.

In the image quality evaluation stage, two objective evaluation technical standards are adopted in the experiment, specifically PSNR and SSIM standards are used for objectively evaluating the reconstruction result of the super-resolution image of 9 test images, subjective evaluation is combined on the basis of objective evaluation to evaluate the image definition and the visual characteristics of a real image, and PSNR and SSIM indexes of different images after the super-resolution image reconstruction is carried out by four algorithms are as follows: the quality of the reconstructed images of the four super-resolution algorithms of the images of different types fluctuates greatly, and the comparison among the four methods shows that the quality of the reconstructed images by the bicubic interpolation method is the worst, the SCSR method and the Self-SR method are higher, and the quality of the reconstructed images by the method is the best. Compared with SCSR and Self-SR methods, the method of the invention has the advantages that the PSNR improvement for landscape and people is generally between 0.19 and 0.32dB, and the restoration effect for artificial scene images with obvious texture inside is generally improved by more than 0.7 dB.

In order to better evaluate the reconstructed image of the super-resolution image, three images with better reconstruction and restoration effects are selected from the three types of images respectively, and the three images are amplified through local areas to present the super-resolution reconstructed image of four algorithms.

FIG. 3 shows the reconstruction effect of the test image building at 2-fold super-resolution using Bicubic, SCSR, Self-SR and the method of the present invention. The three groups of images are presented in a contrast way, the indexes are the same as objective evaluation indexes, and the interior of a reconstructed image obtained through bicubic interpolation is obviously blurred, so that the detailed content of the image is not convenient to recognize; compared with the traditional interpolation method, the SCSR and Self-SR algorithm has certain enhancement on reconstruction effect, the image details are relatively clearer, the recognizable areas in the image are more than the original low-resolution image, and the edge areas still have more saw teeth.

By observing the local amplification of each reconstructed image, the method has very good reconstruction effect on the regular texture in the image, is superior to other three algorithms, and is consistent with objective quality evaluation conclusion. As shown in the figure, the image of the Bicubic algorithm is quite fuzzy, the SCSR and Self-SR algorithms restore the gaps among the floor tiles well but the black edges have sawtooth phenomena, and the restoration of the gaps among the floor tiles by the method is close to the edge of a real image and the edge is quite smooth.

The invention introduces geometric invariance of image blocks on a self-example super-resolution image reconstruction method and provides a self-example learning-guided single-image super-resolution reconstruction method. The main improvements include: firstly, by introducing geometric transformation of image blocks, the retrieval space of internal similar image blocks is expanded, so that more image blocks with higher similarity can be retrieved by a target image block in an image pyramid, high-frequency detail information in a reconstructed image obtained by the algorithm is more sufficient, and the reconstruction effect on the internal texture of the image is better; secondly, the image internal plane information is obtained by adopting a vanishing point detection method, and the retrieval area is constrained by using the local relevance of the image in the process of retrieving similar image blocks, so that the calculation amount of nearest neighbor search retrieval is greatly reduced, and the algorithm efficiency is obviously improved.

Claims

1. The single-image super-resolution reconstruction method guided by self-example learning is characterized in that geometric invariance of image blocks is introduced into similar image block matching to realize a single-image super-resolution image reconstruction algorithm based on example geometric invariance, and in the similar image block matching process, the geometric deformation of the image blocks is used as a loss function to expand the search space of internal image blocks;

the first step, image detection and pyramid construction,

Gathering and then passing the vanishing line

The density distribution maps are paired pairwise to obtain plane information;

in the second step, similar image block matching is performed,

thirdly, the image is amplified in multiple stages and reconstructed,

2. The single-image super-resolution reconstruction method guided by example learning of claim 1 is characterized in that the plane information extraction of the invention adopts a vanishing point detection method, firstly plane information in a two-dimensional image is extracted through image edge detection and vanishing point clustering, and the specific steps are as follows:

using vanishing lines

Parameters representing the plane C, by a transmission transformation matrix B_CPerforming geometric correction on transmission images of real planes to make them mutually correspond in three-dimensional spaceThe parallel line segments are still kept parallel in the imaging image;

3. The method for single-image super-resolution reconstruction from paradigm learning guidance according to claim 2, wherein the planar guidance comprises three parts, which are a coplanar loss function, a line segment direction loss function and a nearest neighbor search loss function;

4. The method of claim 1, wherein in the nearest neighbor structure definition, the vanishing line of the image inner plane obtained by the image inner plane detection and localization is used for the super-resolution reconstruction of the single image guided by the example learning

Deriving the transparency of the corresponding planeMatrix of transform

M(t_i)≈T_i*M(S_i)

representing its center coordinates;

N(t_i,b_i)＝T_i(b_i)M(S_i)。

5. the method of claim 4, which is based on the super-resolution reconstruction method of single image guided by example learningIs characterized in that the similarity loss function adopts an Euclidean distance square sum weighted by RGB space Gaussian to measure the image block M (t) of the target block_i) Similar image block N (t) after geometric transformation_i,b_i) Loss function D of similarity between_similarity(t_i,b_i) The form is as follows:

wherein, the matrix

The scale is represented by a transformation of the scale,

the parameters are scaled as follows:

wherein, the matrix

wherein the miscut transform matrix

The form is as follows:

in summary, in combination with transmission transformation

wherein the parameter b in the geometric transformation_i＝(s_i，C_i)，

Estimation of the parameters determines S, E and F matrices in the affine transformation, and utilizes the target image block M (t)_i) And is optimizedMatching similar image blocks M (S)_i) Coordinate to transmission transformation of

6. The method for single-image super-resolution reconstruction guided by self-example learning of claim 4, wherein an image block scale loss function is added in the image block matching process:

D_measure＝g_measuremin(0，SRF-measure(T_i))

7. The method for reconstructing the super-resolution of the single image guided by the example learning of the invention according to the claim 1, wherein the PatchMatch algorithm is adopted in the similar image block matching process, and the search strategy of the PatchMatch algorithm is redefined according to the algorithm characteristics of the invention;

8. The method of claim 7, wherein the neighborhood forward-backward propagation is based on the correlation of local structures of images, that is, the target image block P and the best matching image block Q are similar image blocks, so that the best matching image block of the image blocks adjacent to P tends to be located in the area adjacent to Q, and the propagation process of approximate nearest neighbor search in PatchMatch algorithm includes: forward propagation and backward propagation;

9. The method of claim 1, wherein the synthesis of the high resolution blocks uses a nearest neighbor search method to obtain the target block M in the low frequency pyramid_A(t_i) Approximate nearest neighbor N of_A(S_i) And geometric transformation T_i(b_i) Extracting corresponding high-resolution version N in the high-resolution image pyramid_B(S_i) Then, the corresponding formula of the high resolution image block in the target image is as follows:

M_B(t_i)＝N_B(s_i)T_i ^-1(b_i)

in the reconstruction process, multi-stage amplification is adopted for super-resolution imageReconstructing, without directly obtaining, a target resolution image J_BFirst obtaining J by zooming in step by step_A+1，J_A+1Inputting a low-resolution upper-level image, and then inputting J_A+1As an input image pair J_A+2Performing reconstruction, and when the resolution improvement scale is 2, making B equal to A + 3;

between each stage of amplification, the iterative back projection method is adopted to carry out degradation processing on the currently output high-resolution image and input the low-resolution image J_BError comparison is carried out, the error is back projected to the current reconstructed image for error correction, and each level of reconstructed image and the input low-resolution image J are ensured_BThe consistent characteristics are maintained.