CN106296586B - Multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode - Google Patents
Multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode Download PDFInfo
- Publication number
- CN106296586B CN106296586B CN201610682571.7A CN201610682571A CN106296586B CN 106296586 B CN106296586 B CN 106296586B CN 201610682571 A CN201610682571 A CN 201610682571A CN 106296586 B CN106296586 B CN 106296586B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- kernel
- fusion
- super
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 123
- 230000033001 locomotion Effects 0.000 claims abstract description 29
- 230000004927 fusion Effects 0.000 claims abstract description 22
- 239000011159 matrix material Substances 0.000 claims description 24
- 238000002156 mixing Methods 0.000 claims description 12
- 230000036961 partial effect Effects 0.000 claims description 9
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 239000000470 constituent Substances 0.000 claims description 2
- 238000005457 optimization Methods 0.000 abstract 1
- 230000008569 process Effects 0.000 description 20
- 230000000694 effects Effects 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 7
- 230000006872 improvement Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 239000004615 ingredient Substances 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000013178 mathematical model Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 241000272194 Ciconiiformes Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000000474 nursing effect Effects 0.000 description 1
- 239000012466 permeate Substances 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
Multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode, the invention belongs to computer vision fields, it mainly solves image procossing precision problem and proposes optimization algorithm, the method of the present invention step includes the multiframe low resolution (low-resolution to input, LR) image sequence implementation Union Movement estimation obtains " combining high-resolution (high-resolution, HR) image I with the method for super-resolution rebuildingjoint"；The multiframe LR image sequence of input is implemented to obtain " kernel regression HR image I based on the method for reconstructing for improving kernel regressionkernel"；With convex combination frame proposed by the present invention, the region of fusion is determined according to airspace fusion criterion, the frequency content for needing to merge is determined according to frequency domain criteria, and global weight determines the weight distribution of the fusion to two methods, it is merged by the HR image that these three aspects obtain upper two step, obtains the HR image of reconstruction to the end.
Description
Technical field
The invention belongs to computer vision fields, are related to image procossing and optimum theory problem, and in particular to a kind of new
Super resolution ratio reconstruction method based on the convex combination for improving kernel regression and Combined estimator.
Background technique
With the intensification of information age, multimedia messages have obtained explosive growth, around people without when without
Quarter is not flooded with various digital informations (such as image, audio, video).And along with image, video technique it is wide
General to apply and reach its maturity, requirement of the people for digital picture, video quality is higher and higher.In actual application scenarios
(such as photo perhaps monitoring video of mobile phone shooting) usually require that compare clearly image or video with for digital picture,
The post-processing and analysis of video, and a concept closely related with image definition is exactly the resolution ratio of image.Resolution ratio
Definition is the ability that imaging device shows the good details of object.So a kind of thinking is directly under imaging device hardware aspect
There are two ways to hand, this improvement resolution ratio.One is the pixel quantities for increasing acquired image, but this method can be brought
For more noise pollutions to reduce signal noise ratio (snr) of image but also will increase acquisition time, second method is to increase chip ruler
It is very little, but expensive cost performance is not high.Therefore, a kind of technology of the promotion image resolution ratio of the alternative above method occurs,
That is image super-resolution rebuilding technology, this technology are relatively easily realized and will not be brought and outer expense, moreover it is possible to be brought good
Good visual effect.This method has many applications in computer vision and field of image processing, and important needing to show
The occasion of details is particularly important, such as the text or other are small of offering a clear explanation from low-quality monitoring or mobile video
When details.
In existing image super-resolution rebuilding method, single frames is broadly classified as according to the quantity of input low-resolution image
Super-resolution rebuilding (Single Frame Super Resolution, SFSR) and multiframe super-resolution rebuilding (Multi-
Frame Super Resolution,MFSR).SFSR technology is by estimating to obtain height from many training set HR images mostly
Frequency ingredient makes up the high-frequency information lost in input LR image, and MFSR technology is the information acquisition by merging all LR images
HR image.The invention belongs to the scopes of MFSR technology.The performance of MFSR technology depends primarily on the accuracy of action reference variable.
MFSR method can be divided into two class of frequency domain method and spatial domain method.Frequency domain method is actually that image interpolation is solved the problems, such as in frequency domain,
Observing and nursing is the shift characteristics based on Fourier transformation, but frequency domain method is only limitted to global translation movement and model, and in frequency domain
Lack data dependence, it is very difficult to include airspace priori knowledge, therefore no longer become research mainstream at present.Spatial domain method is in pixel
On scale, spatial resolution is improved and transformation to pixel, constraint.Main spatial domain method has non-homogeneous interpolation method, repeatedly
For backprojection algorithm, projections onto convex sets, and the method based on Bayesian frame-maximum a posteriori probability method and various methods it is mixed
It closes.
Three during the last ten years, and domestic and foreign scholars conduct in-depth research MFSR method, achieves great successes, including
Discussion to priori knowledge model proposes new method for estimating, analysis and improvement to existing scheme etc..However, multiframe
The reconstruction of low-resolution image is studied and application there is problems:
(1) more accurate effective image registration (i.e. accurate estimation) is needed, registration bring is eliminated and rebuilds mistake
Difference；
(2) more quickly reconstruct is needed, since the method reconstructed based on multiframe is mostly iterative solution, in real-time also
It is to be improved, there is a problem of that the deviation of pixel overall gray value is larger and empty after interpolation reconstruction；
(3) more steady robustness is needed, reduces quantization error or noise to reconstruction image contributions.
Summary of the invention
Goal of the invention of the invention is: in view of the above problems, providing a kind of pair of noise robustness and can be accurately
Estimate the new super resolution ratio reconstruction method based on the convex combination for improving kernel regression and Combined estimator of motion information.
A kind of super resolution ratio reconstruction method based on the convex combination for improving kernel regression and Combined estimator proposed by the present invention is
Utilize the more significant two methods of performance in current MFSR method: method and Union Movement estimation and oversubscription based on kernel regression
The method that resolution is rebuild.Method based on kernel regression is the problem for being difficult to accurately estimate for kinematic parameter between image sequence frame
It proposes, it, which does not need accurate estimation, to carry out super-resolution rebuilding to multiple image.This method is in super-resolution
Reconstruction field has outstanding performance, can realize the denoising of image while interpolation reconstruction.Due to the application scenarios of project
It is monitor video, and monitor video has that resolution ratio is lower and smudgy mostly, so with improved kernel regression
Method carries out super-resolution rebuilding.The method of Union Movement estimation and super-resolution rebuilding has very big in terms of precise motion estimation
Advantage.Mainly estimation and super-resolution rebuilding, the two have the process of sequencing to put to this combined estimation method
It is completed together in a frame, optimizes the result of estimation by the estimated value of super-resolution rebuilding, so as to obtain
Compared to the higher precision of existing motion estimation algorithm, the last effect for promoting super-resolution rebuilding in turn again.Both sides
Method is each advantageous, also respectively there is disadvantage, for the method based on kernel regression, because it does not have motion-estimation step, works as
When the objects in images for needing to rebuild has Large Amplitude Motion, rebuilding precision can be declined.And for combined estimation method
For, it is to have that estimation and super-resolution rebuilding the two processes to interact, which can mutually bring the information of another process,
Limit, so it is not unlimited to the promotion of image super-resolution rebuilding effect, that is, the promotion for rebuilding effect
It is not very significant.
In view of the above problems, can effectively merge both methods, mutually the invention proposes a kind of frame of convex combination
Respective disadvantage is made up, on the other hand mutually promotes respective advantage again.There is object significantly to transport in needing the image rebuild
When dynamic, combined estimation method at this time can provide the advantage in accurate estimation, and rebuild the preferable core of effect
Homing method can also further promote the reconstruction effect of Union Movement estimation method.
New method specific implementation proposed by the present invention includes the following steps:
Step 1: the method for implementing Union Movement estimation and super-resolution rebuilding to the multiframe LR image sequence of input obtains
" joint high-resolution HR image Ijoint"；
Step 2: the multiframe LR image sequence of input being implemented to obtain " kernel regression HR based on the method for reconstructing for improving kernel regression
Image Ikernel"；
Step 3: convex combination frame proposed by the present invention is used, the region of fusion is determined according to airspace fusion criterion, according to
Frequency domain criteria determines the weight distribution for needing the frequency content that merges, and global weight to determine the fusion to two methods,
It is merged by the HR image that these three aspects obtain upper two step, obtains the HR image of reconstruction to the end；Convex combination mode
It can state are as follows:
Icom=(1-A(r,s))·Ikernel+A(r,s)·Ijoint (1)
A(r,s)=α VW(r,s) (2)
Wherein V and W(r,s)Airspace Mixed Zone and the frequency domain mixed frequency blending constituent to two images are determined respectively, entirely
Office parameter alpha ∈ [0,1] is determined to joint HR image IjointWith kernel regression HR image IkernelHow much is mixing.
Since main novel point of the invention is the proposition of convex combination frame, so for two kinds of basic MFSR methods:
Simple introduction is only done based on improvement kernel regression method for reconstructing and Union Movement estimation and the specific implementation of super resolution ratio reconstruction method
It does not do and goes into seriously, the determination of convex combination frame is mainly discussed in detail.
The specific process of multi-frame image super-resolution reconstruction method can substantially be divided into three basic steps: registration, interpolation,
It rebuilds.According to different algorithms, three basic links can carry out simultaneously, can also separate independent realize.In general, multiframe oversubscription
Each low-resolution image is registrated by resolution reconstruction firstly the need of by motion estimation process, recycles frequency domain or airspace weight
It builds algorithm the low-resolution image being registrated permeates panel height image in different resolution.And Union Movement estimation and Super-resolution reconstruction
The main feature of construction method is exactly that estimation link carries out simultaneously with super-resolution rebuilding link, two process iterative estimates,
It interacts, improves motion-estimation precision so as to improve effect is rebuild.
Kernel regression is as a kind of non-parametric homing method for needing not rely on data itself and modeling, due to its stronger spirit
Activity and ductility, are widely used in field of image processing, such as: image interpolation, denoising, super-resolution rebuilding.Based on changing
Into the further improvement that kernel regression method for reconstructing is for classical kernel regression theory.Classical kernel regression reconstruction is built upon multinomial
On the basis of interpolation reconstruction method, only considered the grayscale information of image.On the basis of classical kernel regression is theoretical, image is utilized
Partial structurtes feature have also been proposed the multiframe super-resolution reconstruction method of controllable kernel regression a kind of.Controllable kernel regression method for reconstructing exists
While considering grayscale information, the structural information of image is also incorporated by constructing controllable kernel function, although to a certain extent
The visual quality of image is improved, but the calculating of local covariance matrix is to rebuild to calculate gradient from classical kernel regression,
Computation complexity is high, and there is a problem of that the deviation of pixel overall gray value is larger and empty after interpolation reconstruction.The present invention uses
It is a kind of based on improve kernel regression multiframe super resolution ratio reconstruction method.It uses local structure tensor to replace covariance matrix,
The geometry feature for describing image, can reduce the computation complexity of kernel regression algorithm.Also, in the reconstruction of controllable kernel regression
Image pixel overall gray value deviation is greatly and the problem in cavity, the geometric distance function for constructing image neighborhood pixels correct core letter
Number, so that the image reconstructed has preferable visual quality.
Convex combination frame establish on above two method, using kernel regression method rebuild image have preferable details and
The advantage of combined estimation method precise motion estimation, is subject to the fusion of convex combination mode.It makes up mutually bad existing for two methods
Gesture: kernel regression method lacks exercise estimation, and combined estimation method reconstructed image quality is relatively poor.Convex combination mode relies primarily on
In the reconstruction image I that kernel regression method obtainskernel, and for needing to mix place, the mainly airspace that combination needs is mixed
Close criterion V, frequency-domain frequency ingredient mixed criteria W(r,s)And the determination of global mixed criteria α.The purpose of airspace mixed criteria V exists
In finding out the local for needing two images to mix, that is to say, that kernel regression method, which rebuilds HR image, the place of large error,
There is the place of the object of Large Amplitude Motion in the multiframe LR image rebuild.Briefly, V needs to lock kernel regression method
The appearance certain error region of HR image is mixed with integrated processes HR image.Similarly, frequency-domain frequency ingredient mixed criteria W(r,s)Purpose be to find out the frequency components for needing two images to mix, need exist for using steerable pyramid carried out to image
It decomposes.W(r,s)Task be lock kernel regression method HR image occur the primary bands of the larger loss of signal in frequency domain, also
It is to say that the loss of these band signals needs integrated processes HR image bring information to be made up.For global mixed criteria α,
Still there is the normal picture feature having after mainly guaranteeing two images mixing, that is to say, that so that mixed image is inclined
It is minimum from the degree with natural image.
In conclusion by adopting the above-described technical solution, the beneficial effects of the present invention are:
(1) the invention proposes a kind of convex combination modes of innovation, can obtain the SR reconstruction image of high quality；
(2) due to the determination of three kinds of principles of convexity combination of the present invention, to improve the HR image that kernel regression method is rebuild
Bring the information of loss；
(3) since two kind MFSR methods of the present invention to current mainstream have carried out effective mixing, so that the SR algorithm for reconstructing
There is robustness to interframe Large Amplitude Motion and noise, reconstructed image quality is high.
Detailed description of the invention
It will illustrate by example and with reference to the appended drawing, in which:
Fig. 1 is that algorithm flow of the invention is summarized；
Fig. 2 is the correlation schematic diagram of low-resolution image motion estimation parameter of the invention；
Fig. 3 is the mathematical model schematic diagram of kernel regression method of the invention；
Fig. 4 is the three-dimensional S obel gradient operator template that the present invention uses；
Fig. 5 is that the steerable pyramid algorithm that the present invention uses generates showing for multiple dimensioned multi-direction frequency band applied to input picture
It is intended to (being 3 scales and 3 direction frequency bands in example)；
Fig. 6 is the schematic diagram that image block is mixed according to convex combination mode in the present invention.
Specific embodiment
All features disclosed in this specification or disclosed all methods or in the process the step of, in addition to mutually exclusive
Feature and/or step other than, can combine in any way.
Any feature disclosed in this specification (including any accessory claim, abstract and attached drawing), except non-specifically chatting
It states, can be replaced by other alternative features that are equivalent or have similar purpose.That is, unless specifically stated, each feature is only
It is an example in a series of equivalent or similar characteristics.
Referring to Fig.1, of the invention the specific implementation process is as follows:
Embodiment 1
Process 1 implements combined estimation method and generates high-definition picture Ijoint
(1.1) method of super-resolution rebuilding is firstly the need of motion estimation parameter is obtained, and the data such as image fuzzy parameter are
It can be carried out reconstruction.But since these obtained parameters not can guarantee completely accurately, and the capital of error tiny in parameter
It is amplified in reconstruction process.Therefore, if the build-up effect of parameter error can be tried every possible means to reduce, super-resolution rebuilding
Effect will be improved.So to rebuild super-resolution rebuilding frame as follows for integrated processes:
WhereinIt is the HR image rebuild, HkIt is degenerate matrix,It is the prior information about unknown HR image, Q
(Hk) function adjusts degenerate matrix Hk, so that degenerate matrix is more accurate.
(1.2) combined estimation method is thought not to be completely independent between low-resolution image degenerate matrix, passes through foundation
Relationship between degenerate matrix is as restrictive condition, to guarantee that degenerate matrix is more accurate.The process of reconstruction is divided into two steps:
One step does not consider the factors such as fuzzy noise, Union Movement estimation and data fusion, and the information of all low-resolution images is all melted
It closes in piece image；Second step post-processes fused image.Then the model that degrades of image becomes:
Yk=FkZ+Ek (4)
Wherein YkIt is low-resolution image, FkIt is motion estimation parameter matrix, EkIt is noise.Z represents low-resolution image warp
Image after crossing data fusion:
Z=BkX+σk (5)
Wherein BkIt is fuzzy operator, σkIt is noise.Only consider between low-resolution image and reconstruction image in formula (4)
Then movement relation and noise establish the model of Union Movement estimation and data fusion, solve the blending image in formula (4)
Zk, last high-definition picture X is finally solved according to formula (5).
The model of Union Movement estimation and data fusion includes two, data fit term and degenerate matrix limit entry.Data
Fit term uses robust iterative data fitting function:
The restrictive condition of degenerate matrix is as follows:
WhereinIt is kth frame low-resolution image motion estimation parameter inverse of a matrix matrix.Wherein ε is constant, 0 to 1
Between.In view of the weight of different low-resolution frames is distinguishing, the time gap of this difference and low-resolution image
It is related.Adjacent frame correlation is larger, therefore the degenerate matrix from consecutive frame is more credible, so needing to assign biggish power
Value and frame apart from each other, possible picture material change greatly, degenerate matrix confidence level is slightly smaller.Therefore there is employed herein index letters
Number be used as weight function, when two width low-resolution image time gaps farther out when, assign lesser weight.(7) formula thinks each
Width low-resolution image can obtain the image for needing to rebuild, different low-resolution images after " reversed " estimation
After " reversed " estimation should be close for the approximation of reconstruction image.As shown in Figure 2.
(1.3) according to the analysis of front, the model such as following formula of Union Movement estimation and data fusion is established:
Then the above problem is solved with alternative iteration method, first carries out estimation, obtains F1,…FNInitial estimate, so
After assume F1,…FNIt is constant, withIt is iterated for independent variable:
Wherein, ζ represents the step-length of iteration.According to formulaDetermine the no end covered with clouds of entire iterative process.Wherein,
η is threshold value.After above-mentioned iteration stopping, available fused image Z is established with drag:
(10) formula is solved using gradient descent method, the cut-off condition of iteration are as follows:The X finally obtained
It is the HR image of reconstruction.
Process 2 is implemented to generate high-definition picture I based on the method for reconstructing for improving kernel regressionkernel
Based on the improvement that the method for reconstructing for improving kernel regression is to controllable kernel regression theory, and controllable kernel regression theory can
It traces theoretical in classical kernel regression.Classical kernel regression method, independently of the order N of recurrence, its essence is the part of pixel
Weighting procedure, this local linear filtering characteristic are the inherent limitations of traditional kernel regression.The mathematical model of kernel regression method is illustrated
Figure is as shown in Figure 3.The kernel function of classical kernel regression theory contains only the spatial information of sampled point pixel, does not account for pixel
Influence of the gray value distance to weight.And in real image, usually contain different image texture details, the side of image
The different zones such as edge, flat, angle point reflect the different variation characteristic of data.Therefore, on the basis of classical kernel regression is theoretical,
Propose the controllable kernel regression estimation method that image Self-variation is added.This method not only relies on the space length of sampled point, also
Consider the Gray homogeneity between neighborhood sampled point, allows for controllable kernel regression method in this way and be provided with nonlinear filtering characteristic.
Grayscale information robustness by analyzing local pixel value obtains the partial signal construction of image, this information is incorporated kernel function,
The shape and size of change kernel function that can be adaptive, enable the shape of core and the partial structurtes feature of size and image
Match.
But controllable kernel function theory needs the picture interested to each when calculating the structural information of each pixel
The local neighborhood of vegetarian refreshments does covariance calculating；And the calculating of covariance has the phenomenon that computing repeatedly, calculating process is more complicated,
And it is computationally intensive.Powerful of the structure tensor as the local geometry of analysis image, has been successfully applied figure
The fields such as estimation, the removal picture noise of the field of direction of picture.Structure tensor is one and includes each element neighborhood side in image
To the symmetrical matrix with strength information, direction and the structural strength of partial gradient can be accurately portrayed, and calculation amount is small,
Therefore improved kernel regression method for reconstructing with three-dimensional structure tensor replaces covariance matrix, not only can the structure of picture engraving believe
Breath, distinguishes different structural regions, and can be reduced calculation amount, effectively maintains the detailed information such as the edge of image.
(2.1) since the calculating of directional information in kernel regression method needs to calculate gradient, so being calculated improving kernel regression
Gradient estimation is carried out in method first.The operator of edge detection usually detects figure using the direction template convolved image of image space
The vertical edge and horizontal edge of picture, and edge direction is perpendicular to gradient direction, therefore can be carried out by edge detection operator
Gradient estimation.Sobel operator includes simultaneously gaussian derivative and smooth characteristic, therefore to making an uproar in the presence of low-resolution video
Sound has robustness, will not because of noise interference and adverse effect caused to subsequent result, and have and quickly calculate gradient
Characteristic.So use 3D-Sobel operator and multiple image sequence frame convolutional calculation gradient value can be in improving kernel regression method
It is effective to reduce the time for calculating big moment matrix.Fig. 4 is used 3D-Sobel operator template.
(2.2) kernel regression rebuilds the technology being built upon on the basis of gradient and interpolation, after obtaining frame sequence gradient, needs
Calculate structure tensor direction come obtain adapt to picture structure interpolation kernel, and structure tensor matrix be from partial gradient derive and
Come, the feature vector and characteristic value of structure tensor can describe the energy principal direction of specific neighborhood of pixel points, in sampling location P
The three-dimensional structure tensor of point is as follows:
Wherein,The gradient of representative image, I representing input images, subscript x, y, t respectively represent three gradient directions, and W is
Partial analysis window, three-dimensional structure tensor S (P) are symmetrical matrixes, and after characteristic value solves, three-dimensional structure tensor indicates as follows:
S (P)=λ1e1e1 T+λ2e2e2 T+λ3e3e3 T (12)
Wherein, λ1≥λ2≥λ3>=0 is the characteristic value of matrix being arranged in decreasing order, e1, e2, e3It is their corresponding Characteristic Vectors
Amount, the intensity of their representative image partial structurtes and direction.Then improved kernel function are as follows:
XiNoise samples point is represented, X is the interested sampled point to be estimated, and h is referred to as global smoothing parameter.
(2.3) in reconstruction process, gray value of image deviation is excessive, using the pixel interdependence principle of image, by image slices
The neighbouring geometric distance function of element, which is inserted into above-mentioned improved kernel function, can inhibit the generation of this phenomenon, and this function calculates
Value out can save in advance.Adjacency function are as follows:
α is correction factor, and the mutually multiplied controllable kernel function expression formula finally improved of formula (13):
Final solution form is as follows:
Wherein,Represent the pixel value of the point-of-interest to be estimated.Y be each sampled point gray value column to
Amount, XXRepresent each rank distance matrix of sampled point, e1Be header element be 1, remaining be 0 column vector.WXFor improved controllable kernel function
KHThe diagonal matrix of ().
WX=diag [KH(X1-X),KH(X2-X),...,KH(XP-X)]×diag[K(X1),K(X2),...,K(XP)]
(17) (2.4) pass through and analyze above, and the key step of improved kernel regression multiframe super-resolution rebuilding is summarized as follows:
(1) multiple image sequence is inputted
(2) gradient of each interested pixel point is calculated using 3D sobel operator；
(3) three-dimensional partial gradient structure tensor S (P) is calculated using formula (11), acquires three-dimensional structure directional information；
(4) the geometric distance functional value of the neighborhood territory pixel of amendment kernel function is calculated using formula (14)；
(5) solution for finding out step (3) and (4) substitutes into formula (15), obtains improved kernel function value KH(·)；
(6) the kernel function value K for obtaining step (5)H() substitutes into formula (17) and obtains WX；
(7) W for obtaining step (6)XSubstitution formula (16) obtains final output image Ikernel。
Process 3 implements Frank-Wolfe algorithm of the invention and generates high-definition picture Icom
Convex combination frame is established on above two method, using respective advantage, the shortcomings that making up other side, is obtained more
The reconstruction effect of high quality.Referring to Fig. 4, I is generatedcomThe specific implementation process is as follows:
(3.1) airspace mixed criteria V is determined:
In the high-definition picture I for calculating the generation of kernel regression methodkernelWhen, it needs to calculate sense with 3D sobel operator emerging
The gradient of interesting pixel can obtain the gradient of the area-of-interest in the multiframe LR image sequence of input in the process, by
The place of the significant changes of gray value of image is reflected to a certain extent in the gradient value of image.So if LR to be reconstructed schemes
There are the wings acutely agitated in the object of strenuous exercise, such as penguin picture in area-of-interest as in, then every frame image
The gradient value of corresponding region has certain deviation compared to other static regions.Therefore, gradient is being calculated using kernel regression method
When value, the variance of multiple gradient values of the same ROI pixel of multiframe LR image can be used as this region with the presence or absence of significantly transporting
One Measure Indexes of animal body.The variance of the correspondence gradient value of the LR image ROI region of all frames is calculated, and normalizes to 0,
Between 1, a variance map (variance map) V has thus just been obtained.The relatively high region of variance map intermediate value is corresponding
It is bigger to there is the probability substantially moved in region, therefore is just added with greater need for convex combination frame proposed by the present invention more containing essence
The information of true estimation also further illustrates moving party and poorly schemes the reasonability that V this concept mixes in airspace.Method exists
When airspace is implemented, choosing V value in variance map is more than that the region of certain threshold value is mixed, and other regions remain unchanged.
(3.2) frequency-domain frequency ingredient mixed criteria W is determined(r,s):
Frequency-domain frequency ingredient mixed criteria W(r,s)Purpose be to determine the primary bands that are mixed in frequency domain.Due to
Image, the high-resolution rate image I that the present invention obtains improvement kernel regression method are handled in frequency domainkernelUsing controllable golden word
Tower (steerable pyramid) algorithm decomposes frequency domain, obtains image IkernelDifferent scale and different directions frequency band point
Solution.Steerable pyramid algorithm can be provided by the decomposition of multiple dimensioned multi-direction frequency band to image joint localization (in space or
Frequency domain) expression.As shown in Figure 5.The direction number of frequency bands r ∈ (0, R) decomposed is represented with R, S represents the scale quantity s decomposed
∈ (0, S), on different scale S, the decomposition result of available R different directions band is more accurately selected convenient for us
Mixed direction frequency band is needed on different scale.By variance map V obtained in the previous step, the area for having Large Amplitude Motion to change is chosen
Implement controllable pyramid decomposition and obtain the decomposition of R direction frequency band of S scale, then calculate these image districts in domain (assuming that V > 0.5)
Average energy of the domain under the frequency band of different scale direction is distributed with one under the frequency band of different scale direction according to region energy
Important conclusion: the loss of signal occurs most universal in the frequency band containing higher-energy.And these contain the frequency band of higher-energy
Preferably corresponding to a certain partial-band in R direction band decomposition under different scale S.Introduce new variableIt indicates
With the part frequency band of higher-energy under scale S.λ value range is to control the frequency chosen from total frequency band number R between 0 to 1
The number of band.By analysis above, for available at the location of pixels x in ROI region:
W in above formula(r,s)Selection can effectively make two images IkernelAnd IjointAlong image IkernelIn need most volume
Those of external information frequency band is mixed.The mixing schematic diagram in block region is as shown in Figure 6.
(3.3) global hybrid parameter α is determined:
The main function of global hybrid parameter α is to ensure that mixed two images are the width for meeting natural image property
True image.This needs is analyzed by the statistical property that natural image decomposes band logical: natural image is for bandpass filtering
The skirt response of device is height non-gaussian.And the peak value (Kurtosis) of response can be used for the departure degree of Gauss model
To measure.And actually also there is the peak value of investigation display natural image to be kept approximately constant over different frequency bands really.Certain point
The peak value of cloth is defined as:
Wherein μ4It is the Fourth-order moment for being distributed mean value, σ is the standard deviation of distribution.By this definition it is found that Gaussian Profile
Peak value is 0.So the selection of optimum alpha should make last mixing HR image IcomPeak value on different frequency bands has minimum
Variation.That is α should meet:
WhereinIt is last mixing HR image IcomFor comprising S scale and each scale contains R direction
The peak value of the skirt response distribution of the bandpass filter of frequency band.It is the mean value of the peak value on all frequency bands.Above formula uses
The fminsearch function of MATLAB is readily available solution.Finally implement mixing, (2) can be brought into according to the analysis of two step of front
Formula obtains A(r,s)It brings (1) formula into again and obtains reconstruction HR image I to the endcom。
The invention is not limited to specific embodiments above-mentioned.The present invention, which expands to, any in the present specification to be disclosed
New feature or any new combination, and disclose any new method or process the step of or any new combination.
Claims (3)
1. the multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode, which is characterized in that including following step
Suddenly,
Step 1 combines the multiframe LR image sequence implementation Union Movement estimation of input with the method for super-resolution rebuilding
High-resolution HR image Ijoint；
Step 2 implements the multiframe LR image sequence of input to obtain kernel regression HR image based on the method for reconstructing for improving kernel regression
Ikernel；
Step 3, the region that fusion is determined according to airspace fusion criterion, the frequency content for needing to merge is determined according to frequency domain criteria,
The weight distribution to fusion is determined further according to global weight；
Step 4, according to convex combination model, it is high to joint according to the region of fusion, the frequency content of fusion, the weight distribution of fusion
Resolution ratio HR image IjointWith kernel regression HR image IkernelMerged the HR image rebuild；
Step 3 includes the following steps,
The variance of step 3.1, multiple gradient values of corresponding multiframe LR interesting image regions, is normalized between 0 to 1 simultaneously
As the element of variance map, preset threshold judges all elements V (i, j) greater than threshold value, so that it is determined that the area of fusion
Domain；
Step 3.2, using the area-of-interest of steerable pyramid algorithm decomposition step 3.1, find out in each area-of-interest and have
There is the main band of higher-energy, to determine the frequency content for needing to merge；
The weight distribution of step 3.3, the statistical property decomposed according to natural image to band logical, fusion should be chosen so that super-resolution
The peak value of reconstruction image on different frequency bands has the smallest variation, to determine the weight distribution to fusion；
Step 4, convex combination model is
Icom=(1-A(r,s))·Ikernel+A(r,s)·Ijoint
A(r,s)=α VW(r,s)
Wherein, V and W(r,s)Determination respectively corresponds joint high-resolution HR image IjointWith kernel regression HR image IkernelAirspace it is mixed
The blending constituent in region and frequency domain mixed frequency is closed, global parameter α ∈ [0,1] determines joint HR image IjointScheme with kernel regression HR
As IkernelMixed proportion.
2. the multi-frame low resolution image super resolution ratio reconstruction method according to claim 1 based on convex combination mode,
It being characterized in that, step 1 includes the following steps,
Step 1.1, the model that degrades according to image, utilize the movement between multiframe LR image sequence and super-resolution rebuilding image
Relationship and noise, establish the model of Union Movement estimation and data fusion, and alternating iteration goes out blending image；
Step 1.2, using super-resolution rebuilding frame, substitute into blending image, iteration goes out joint high-resolution HR image Ijoint。
3. the multi-frame low resolution image super resolution ratio reconstruction method according to claim 1 based on convex combination mode,
It being characterized in that, step 2 includes the following steps,
Step 2.1, input multiframe LR image sequence
Step 2.2, the gradient that each interested pixel point is calculated using 3D sobel operator；
Step 2.3 calculates three-dimensional partial gradient structure tensor S (P), acquires three-dimensional structure directional information；
Step 2.4 calculates the geometric distance functional value for correcting the neighborhood territory pixel of kernel function；
Step 2.5, the solution for finding out step 2.3 and step 2.4, which substitute into, improves kernel function and by its interpolation adjacent to geometric distance letter
The product for the adjacency function that number obtains, obtains improved kernel function value K ()；
Step 2.6 constructs diagonal matrix W using kernel function value K ()X；
Step 2.7, according to diagonal matrix WX, sampled point is set, kernel regression HR image I is obtainedkernel。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610682571.7A CN106296586B (en) | 2016-08-18 | 2016-08-18 | Multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610682571.7A CN106296586B (en) | 2016-08-18 | 2016-08-18 | Multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106296586A CN106296586A (en) | 2017-01-04 |
| CN106296586B true CN106296586B (en) | 2019-07-05 |
Family
ID=57679793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610682571.7A Active CN106296586B (en) | 2016-08-18 | 2016-08-18 | Multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106296586B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106846250B (en) * | 2017-01-22 | 2020-05-22 | 宁波星帆信息科技有限公司 | Super-resolution reconstruction method based on multi-scale filtering |
| CN107578017B (en) | 2017-09-08 | 2020-11-17 | 百度在线网络技术（北京）有限公司 | Method and apparatus for generating image |
| CN108827249B (en) * | 2018-06-06 | 2020-10-27 | 歌尔股份有限公司 | Map construction method and device |
| CN109493280B (en) * | 2018-11-02 | 2023-03-14 | 腾讯科技（深圳）有限公司 | Image processing method, device, terminal and storage medium |
| CN110236584B (en) * | 2019-06-25 | 2022-04-12 | 新里程医用加速器(无锡)有限公司 | Dual-energy spectrum cone-beam CT system, control method, reconstruction method and device |
| CN110660051B (en) * | 2019-09-20 | 2022-03-15 | 西南石油大学 | Tensor voting processing method based on navigation pyramid |
| CN110856048B (en) * | 2019-11-21 | 2021-10-08 | 北京达佳互联信息技术有限公司 | Video repair method, device, equipment and storage medium |
| CN112082915B (en) * | 2020-08-28 | 2024-05-03 | 西安科技大学 | Plug-and-play type atmospheric particulate concentration detection device and detection method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101226634A (en) * | 2008-01-30 | 2008-07-23 | 哈尔滨工业大学 | Method for reestablishment of single frame image quick super-resolution based on nucleus regression |
| CN102194222A (en) * | 2011-04-26 | 2011-09-21 | 浙江大学 | Image reconstruction method based on combination of motion estimation and super-resolution reconstruction |
-
2016
- 2016-08-18 CN CN201610682571.7A patent/CN106296586B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101226634A (en) * | 2008-01-30 | 2008-07-23 | 哈尔滨工业大学 | Method for reestablishment of single frame image quick super-resolution based on nucleus regression |
| CN102194222A (en) * | 2011-04-26 | 2011-09-21 | 浙江大学 | Image reconstruction method based on combination of motion estimation and super-resolution reconstruction |
Non-Patent Citations (3)
| Title |
|---|
| A New Method for Multi-Frame Super-Resolution Reconstruction;Quan Xiao, et al.;《2009 International Joint Conference on Computational Sciences and Optimization》;20090426;全文 * |
| Image and Video Restorations via Nonlocal Kernel Regression;Haichao Zhang, et al.;《IEEE TRANSACTIONS ON CYBERNETICS》;20130630;第43卷(第3期);全文 * |
| 多帧图像超分辨率重建算法研究;王会鹏;《中国优秀硕士学位论文全文数据库 信息科技辑》;20120315(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106296586A (en) | 2017-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106296586B (en) | Multi-frame low resolution image super resolution ratio reconstruction method based on convex combination mode | |
| Charbonnier et al. | Two deterministic half-quadratic regularization algorithms for computed imaging | |
| Trinh et al. | Novel example-based method for super-resolution and denoising of medical images | |
| US7831088B2 (en) | Data reconstruction using directional interpolation techniques | |
| DE112004000393B4 (en) | System and method for tracking a global shape of a moving object | |
| Zhang et al. | Image super-resolution reconstruction based on sparse representation and deep learning | |
| CN107221013A (en) | One kind is based on variation light stream estimation lung 4D CT Image Super Resolution Processing methods | |
| CN107341844A (en) | A kind of real-time three-dimensional people's object plotting method based on more Kinect | |
| Alpers et al. | Geometric reconstruction methods for electron tomography | |
| CN112184549B (en) | Super-resolution image reconstruction method based on space-time transformation technology | |
| CN107330854A (en) | A kind of image super-resolution Enhancement Method based on new type formwork | |
| An et al. | Image super-resolution reconstruction algorithm based on significant network connection-collaborative migration structure | |
| CN106683049A (en) | Reconstruction method of the image super-resolution based on the saliency map and the sparse representation | |
| CN107767342B (en) | Wavelet transform super-resolution image reconstruction method based on integral adjustment model | |
| Gu et al. | Customized 3D digital human model rebuilding by orthographic images-based modelling method through open-source software | |
| Liu et al. | SAR image denoising based on patch ordering in nonsubsample shearlet domain | |
| Cho et al. | Example-based super-resolution using self-patches and approximated constrained least squares filter | |
| Liu et al. | Lunar DEM Super-resolution reconstruction via sparse representation | |
| Yang et al. | Content-adaptive mesh modeling for fully-3D tomographic image reconstruction | |
| Song et al. | Adaptive interpolation scheme for image magnification based on local fractal analysis | |
| CN113610863A (en) | Multi-exposure image fusion quality evaluation method | |
| Xie et al. | Joint reconstruction and calibration using regularization by denoising | |
| Zhao et al. | Multi-frame image super-resolution based on regularization scheme | |
| CN104881842B (en) | A kind of image super-resolution method based on picture breakdown | |
| Biswal et al. | A parallel approach for affine transform of 3D biomedical images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information |
Inventor after: Gao Jianbin Inventor before: Gao Jianbin Inventor before: Tang Huan |
|
| CB03 | Change of inventor or designer information | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220222 Address after: 610041 No. 1677, north section of Tianfu Avenue, Wuhou District, Chengdu, Sichuan Patentee after: Chengdu financial dream workshop Investment Management Co.,Ltd. Address before: 611731, No. 2006, West Avenue, Chengdu hi tech Zone (West District, Sichuan) Patentee before: University of Electronic Science and Technology of China |
|
| TR01 | Transfer of patent right |