CN106254720A - A kind of video super-resolution method for reconstructing based on associating regularization - Google Patents
A kind of video super-resolution method for reconstructing based on associating regularization Download PDFInfo
- Publication number
- CN106254720A CN106254720A CN201610567213.1A CN201610567213A CN106254720A CN 106254720 A CN106254720 A CN 106254720A CN 201610567213 A CN201610567213 A CN 201610567213A CN 106254720 A CN106254720 A CN 106254720A
- Authority
- CN
- China
- Prior art keywords
- video
- resolution
- regularization
- super
- associating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/142—Edging; Contouring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/20—Circuitry for controlling amplitude response
- H04N5/205—Circuitry for controlling amplitude response for correcting amplitude versus frequency characteristic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
- H04N5/213—Circuitry for suppressing or minimising impulsive noise
Abstract
The invention discloses the method for reconstructing of a kind of video super-resolution based on associating regularization.Comprise the following steps: set up low resolution observing and nursing；By least-square methods based on regularization, set up the cost function of a general video reconstruction；Build based on TV (CTV) regularization term compensated and give the adaptive weighted coefficient of regional space to reduce the harmful effect that registration error is brought；Build several non-local low rank (MNLR) regularization term；Set up the cost function that present invention video super-resolution based on associating regularization is rebuild；Solve cost function with Split Bregman alternative manner, reconstruct high-resolution video.The frame of video that video super-resolution method for reconstructing of the present invention is rebuild out has abundant marginal information, and almost without sawtooth effect, can be seen that from video reconstruction frames the ability of this method suppression noise is also fine, objective evaluation parameter has the highest reference value.So the present invention is a kind of effective video super resolution ratio reconstruction method.
Description
Technical field
The present invention designs a kind of video super-resolution method for reconstructing based on associating regularization, and especially this regularization term was both
Consider the dependency in frame of video, it is also contemplated that the dependency of video interframe, it is possible to keeping frame of video edge details letter
While breath, well suppress noise, belong to digital image processing field.
Background technology
Obtaining during video, due to atmospheric perturbation, lack sampling, system noise and optics and motion blur etc. because of
The impact of element, causes the video quality degradation obtained, and resolution reduces.Along with people to the requirement of the definition of video increasingly
Height, the resolution how improving video becomes problem demanding prompt solution.Super-resolution rebuilding technology is because need not high hardware
Cost just can have a good reconstruction effect and by the concern widely of domestic and international researcher, be in recent years image procossing and
The study hotspot in the fields such as computer vision.Video super-resolution utilizes the similar continuous low resolution differed again to regard when rebuilding
Frequently frame, by the redundancy between frame of video, suppression noise, reduces blooming acquisition high-resolution video frame.Frame of video
Super-resolution reconstruction is made to be built into ill-conditioning problem by the deficiency of various noises or redundancy, therefore by regularization term constraint height
The method of image in different resolution solution space can solve this problem.Tikhonov regularization term realizes simple, but this regularization
Having obscured image border while constraint noise, this method can make the image of reconstruction the most smooth.Based on full variation (TV)
It is the prior image model the most generally used with the regularization constraint of bilateral full variation (BTV), can remove due to them and make an uproar
Reconstruct the image of sharpening while sound, preserve image edge details information, but when video is by bigger influence of noise,
The best at flat site treatment effect, easily produce pseudo-edge, edge region the most easily produces sawtooth effect.
Summary of the invention
The present invention is directed to the deficiency of above regularization, it is proposed that a kind of associating regularization method, by TV based on compensation
(CTV) combine with several non-local low rank regularization term, make full use of the different advantages of two kinds of regularizations, in view of regarding
Frequently while in-frame correlation, explore the dependency of video interframe, regularization term introduces the redundancy between adjacent video frames
Information.Simultaneously take account of and CTV registrates the harmful effect that residual error is brought, CTV is devised the weighting of regional space information filter
Coefficient, reduces the registration residual error negative influence to the reconstruction of video.Algorithm for reconstructing is made to keep regarding while eliminating noise
Frequently the edge detail information of frame.
The present invention proposes video super-resolution method for reconstructing based on associating regularization, mainly comprises following operating procedure:
(1) low resolution observing and nursing is set up；
(2) by least-square methods based on regularization, the cost function of a general video reconstruction is set up；
(3) build TV (CTV) regularization term based on motion compensation and give the adaptive weighted coefficient of regional space to subtract
The harmful effect of little registration error；
(4) several non-local low rank (MNLR) regularization term is built；
(5) according to step (3) and the regularization term in step (4), set up present invention video based on associating regularization and surpass
The cost function of resolution reconstruction；
(6) solve cost function with Split-Bregman alternative manner, reconstruct high-resolution video.
Accompanying drawing explanation
Fig. 1 is the block diagram of present invention video super-resolution based on associating regularization method for reconstructing
Fig. 2 is the present invention with four kinds of methods to video " Mobile " the comparison diagram of reconstructed results of the 15th frame
Fig. 3 is the present invention with four kinds of methods to video " Grandma " the reconstructed results comparison diagram of the 15th frame
Fig. 4 is the present invention with four kinds of methods to video " Akiyio " the reconstructed results frame comparison diagram of the 15th frame
Detailed description of the invention
Video reconstruction method based on associating regularization mainly comprises the steps of
(1) low resolution observing and nursing is set up；
(2) by least-square methods based on regularization, the cost function of a general video reconstruction is set up；
(3) build TV (CTV) regularization term based on motion compensation and give the adaptive weighted coefficient of regional space to subtract
The harmful effect of little registration error；
(4) several non-local low rank (MNLR) regularization term is built；
(5) according to step (3) and the regularization term in step (4), set up present invention video based on associating regularization and surpass
The cost function of resolution reconstruction；
(6) cost function is solved with Split-Bregman alternative manner.
Concrete, in described step (1), the low resolution observing and nursing of frame of video is:
yk=DBfk+N 1≤k≤K, (1)
Wherein yk, fkBeing respectively kth frame low resolution video frame and kth frame high-resolution video frame, D is the down-sampling factor,
B is fuzzy factor, and N is random noise, and K is video frame number.
In described step (2), introduce suitable prior information when video super-resolution is rebuild and regard to construct high-resolution
The constraint of frequency, is guided the super-resolution rebuilding of video, can be realized by least-square methods based on regularization, formula
As follows:
Wherein, Section 1 is data fidelity items, and Section 2 is regularization term, and λ is the weight coefficient of regularization.
In described step (3), structure is based on TV (CTV) regularization term compensated.The method includes the steps of:
The first step, it is assumed that to each frame fk, haveWherein Qk-1, Bk+1Represent respectively with fkFor
Reference frame is to fk-1,fk+1Operation of motion compensation, the present invention uses optical flow method to carry out estimation；
Second step, is arranged on gradient field by the residual error between the consecutive frame after reference frame and motion compensation, and formula is as follows:
Wherein G represents gradient operation, G=[Gh,Gv],GhWith GvRepresent difference operation both horizontally and vertically respectively.
Residual error is arranged on gradient field by the present invention, rather than as variation complete with tradition (TV), original image is arranged on ladder
Degree territory, if it is enough accurate to be because estimation, residual error will be more sparse than original image.
Owing to CTV relies on the difference between the content of frame of video and consecutive frame, so it is insensitive to edge.But,
There is the local motion of complexity between consecutive frame, accurate motion registration is difficult to obtain, and therefore registration residual error will affect and regard
The reconstructed results of frequency.In view of s in step (3)kAccuracy, to each s in gradient fieldkAdd a regional space adaptive
Answering weight so that CTV is converted into CATV, formula is as follows:
Wherein step WkSolution procedure as follows；
The first step, needs to extract spatial structural form, owing to for radio-frequency component, second dervative can distinguish image border
With noise region, define so second derivative-based curvature difference arithmetic operators to be applied to regularization weight coefficient
In.Curvature difference at pixel j point is:
Cj=| | eηη|-|eεε||
Wherein, ex,eyIt is respectively first derivative horizontally and vertically, e at pixelxx,exy,eyyAt pixel
Second dervative.
Second step, calculates the partial weight at pixel, and formula is as follows:
Wherein, τ is constant, controls parameter intensity.
3rd step, utilizes clustering algorithm, by wjDivide M class, then image is divided into discontinuous region, has similarity weight
The pixel of weight is then divided into identical region, and the regularization intensity in each region is by of a sort cluster centre rm(m=1 ...,
M) control.
4th step, calculates final weight equation:
Wherein Γ1,Γ2The smooth region of difference representative image and details area.
It should be noted that weight Wk(k=1 ..., K) design for residual error in gradient field, rather than frame of video.
Can be seen that edge region weight is less, therefore there is less regularization constraint, reduce registration error pair
The impact rebuild, maintains the detailed information of reconstruction.At flat site, weight is relatively big, effectively suppresses noise.
In described step (4), the step building several non-local low rank (MNLR) regularization term is:
The first step, frame of video is divided into overlap block, it is assumed that will be in kth frame, centered by pixel j, size be n ×
The image block of n represents becomes pk,j, centered by j, determine search window；
Second step, the optical flow field in recycling step (3), guide search window in (k-1) frame and (k+1) frame
Center position.Accordingly, it is determined that the center position of the search window in kth frame, it is possible to use in optical flow field this
The motion vector of central point, finds out the center position of corresponding search window in consecutive frame；
3rd step, all candidate blocks and reference block during in calculating (k-1) frame, k frame and (k+1) frame, search window is interior
pk,jBetween l2Difference, finds out pk,jThe N of blockpIndividual similar block；
4th step, is grouped similar block, each structure matrix Xj,k, every string of this matrix represents a similar block.Root
According to similar characteristic, this matrix has low-rank characteristic, but due to the interference of noise, this characteristic is destroyed；
5th step, builds MNLR regularization term, and formula is as follows:
Wherein, Sk,jF represents the reference block sought in (k-1) frame, k frame and (k+1) frame in k frame centered by pixel j
Similar block, and Xj,k=Sk,jF, Zk,jIt it is approximation S of a low-rankk,jThe matrix of F, λ is a balance parameter.
Cost function in step (5) is:
Wherein α, β, λ regularization parameter, use and replace Rank (Z with minor functionk,j):
δ is a constant the least, I representation unit matrix, then cost equation change in order to:
The step using the optimization algorithm of the super-resolution rebuilding of associating regularization term in step (7) is as follows:
The first step, increases a variable d so thatCost equation is converted into for below equation:
Second step, uses Split-Bregman iterative algorithm to solve problem above, then cost function can be decomposed into following
Four subproblems:
3rd step, solves the subproblem of above four respectively:
1 subproblem solving Z:
Zn+1=U (Σ-λ diag (θn))+VT (16)
Wherein, It isThe l eigenvalue, Sk,jFnCan carry out Eigenvalues Decomposition is U Σ VT, and
And (x)+=max (x, 0).
2 subproblems solving F:
Solve by conjugate gradient method (CG) herein.
3 subproblems solving d:
Wherein,
In order to effectiveness of the invention is described, the present invention will choose 3 test videos and test, i.e. " Mobile "
" Grandma " and " Akiyo ".First by video size 5 × 5, variance be 1 fuzzy core obscure, under then carrying out
Sampling, is eventually adding the zero-mean Gaussian noise that variance is 3, generates low-resolution video.Bicubic interpolation is chosen in control methods
Bicubic and 3 kinds of video super-resolution method for reconstructing.Experimental result is as shown in Fig. 2, Fig. 3 and Fig. 4, and (b) arrives (f) respectively
Represent the frame of video that bicubic interpolation image, method 1, method 2, method 3 and the present invention reconstruct.
Three kinds of video super-resolution algorithm for reconstructing are respectively as follows:
Method one: software algorithm, Infognition Video Enhancer [EB/OL], vol.5, no.1,2014.
http://www.infognition.com/videoenhancer/.
The method that method two: Yuan et al. proposes.List of references " Q.Yuan, L.Zhang, H.Shen. " Regional
spatially adaptive total variation super-resolution with spatial information
filtering and clustering."IEEE Transactions on Image Processing,vol.22,no.6,
pp.2327-2342,2013.”。
The method that method three: Turgay et al. proposes.List of references " E.Turgay, G.B.Akar. " Texture and
edge preserving multiframe super-resolution."Image Processing,IET,vol.8,no.9,
pp.499-508,2014.”。
The content of contrast experiment is as follows:
Experiment 1 is to enter the 10 to 20th frame of video " Mobile ", " Grandma " and " Akiyo " by Bicubic method
2 times of super-resolution rebuildings of row, shown in experimental result such as Fig. 2 (b), Fig. 3 (b) and Fig. 4 (b).
Experiment 2 is, by method 1, the 10 to 20th frame of video " Mobile ", " Grandma " and " Akiyo " is carried out 2 times
Super-resolution rebuilding, shown in experimental result such as Fig. 2 (c), Fig. 3 (c) and Fig. 4 (c).
Experiment 3 is, by method 2, the 10 to 20th frame of video " Mobile ", " Grandma " and " Akiyo " is carried out 2 times
Super-resolution rebuilding, shown in experimental result such as Fig. 2 (d), Fig. 3 (d) and Fig. 4 (d).
Experiment 4 be usage 3 the 10 to 20th frame of video " Mobile ", " Grandma " and " Akiyo " is carried out 2 times surpass
Resolution reconstruction, shown in experimental result such as Fig. 2 (e), Fig. 3 (e) and Fig. 4 (e).
Experiment 5 is, by the present invention, the 10 to 20th frame of video " Mobile ", " Grandma " and " Akiyo " is carried out 2 times
Super-resolution rebuilding, shown in experimental result such as Fig. 2 (f), Fig. 3 (f) and Fig. 4 (f).
Several groups of contrast contrast experiments show, the frame of video arrived that Bicubic rebuilds is the fuzzyyest, and there is sawtooth effect at edge,
In frame of video, noise is the most serious；The method that method 1 is rebuild the frame of video of reconstruction still can not be overcome the fuzzyyest and
The shortcoming that noise is serious；Rebuild, from method 2, the frame of video obtained and can be seen that method 2 denoising effect is relatively good, but at edge
Part still has sawtooth effect；The frame of video that method 3 reconstruction obtains is in edge sharpening, but noise phenomenon is the most serious；Use
The present invention rebuilds the frame of video obtained, the result being obtained in terms of no matter being also to maintain edge in terms of removing noise.
For objective appraisal effectiveness of the invention, with Y-PSNR (PSNR:Peak Signal to Noise
And structural similarity (SSIM:Structure Similarity Index) is as objective evaluation evaluation criterion, table one Ratio)
Illustrate the average PSNR and average SSIM of various algorithm.
Table one illustrates that, to two kinds of objective criterions, the present invention has the highest value, for video " Mobile ",
The average PSNR value method 2 of " Grandma " and " Akiyo " present invention exceeds 1.09dB, 1.63dB and 2.01dB respectively,
Average SSIM value ratio method 2 exceeds 0.0303,0.0121 and 0.0192 respectively.
In sum, the present invention has obvious advantage at subjective vision or in objective evaluation standard.Therefore originally
Invention is a kind of effective video super resolution ratio reconstruction method.
Table one
Claims (3)
1. a video super-resolution method for reconstructing based on associating regularization, mainly comprises following operating procedure:
Step one: set up low resolution observing and nursing；
Step 2: by least-square methods based on regularization, set up the cost function of a general video reconstruction；
Step 3: build TV (CTV) regularization term based on motion compensation and give the adaptive weighted coefficient of regional space to subtract
The harmful effect of little registration error；
Step 4: build several non-local low rank (MNLR) regularization term；
Step 5: according to step 3 and the regularization term in step 4, set up present invention video based on associating regularization oversubscription
The cost function that resolution is rebuild；
Step 6: solve cost function with Split-Bregman alternative manner, reconstruct high-resolution video.
A kind of video super-resolution based on associating regularization the most according to claim 1 is rebuild, it is characterised in that step
Three, TV regularization term based on motion compensation is combined by step 4 with several non-local low rank (MNLR) regularization term as
The prior-constrained of video super-resolution reconstruction carries out super-resolution rebuilding.
A kind of based on associating regularization video super-resolution described in requirement the most according to claim 1 is rebuild, and its feature exists
Being difficult to obtain due to accurate estimation in step 3, registration error cannot be avoided in CTV, therefore to believe based on space
Breath filtering reduces, with the weight coefficient of classification, the negative influence that registration error in CTV is brought, and the region high in registration error is joined
Put little weight, reduce the regularization constraint of CTV, in the weight that the region configuration that registration error is low is big, reconstruct high score with this
The video of resolution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610567213.1A CN106254720B (en) | 2016-07-19 | 2016-07-19 | A kind of video super-resolution method for reconstructing based on joint regularization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610567213.1A CN106254720B (en) | 2016-07-19 | 2016-07-19 | A kind of video super-resolution method for reconstructing based on joint regularization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106254720A true CN106254720A (en) | 2016-12-21 |
| CN106254720B CN106254720B (en) | 2019-07-09 |
Family
ID=57613338
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610567213.1A Active CN106254720B (en) | 2016-07-19 | 2016-07-19 | A kind of video super-resolution method for reconstructing based on joint regularization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106254720B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107194889A (en) * | 2017-05-11 | 2017-09-22 | 西安电子科技大学 | The bilateral full variational regularization image denoising method of piecemeal |
| CN108416371A (en) * | 2018-02-11 | 2018-08-17 | 艾视医疗科技成都有限公司 | A kind of diabetic retinopathy automatic testing method |
| CN108765322A (en) * | 2018-05-16 | 2018-11-06 | 上饶师范学院 | Image de-noising method and device |
| CN110208211A (en) * | 2019-07-03 | 2019-09-06 | 南京林业大学 | A kind of near infrared spectrum noise-reduction method for Detecting Pesticide |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103971354A (en) * | 2014-05-19 | 2014-08-06 | 四川大学 | Method for reconstructing low-resolution infrared image into high-resolution infrared image |
| CN104463793A (en) * | 2014-11-12 | 2015-03-25 | 合肥工业大学 | Video super-resolution reestablishing method and system based on sparse representation and vector continued fraction interpolation under polar coordinates |
| US20170011710A1 (en) * | 2014-01-21 | 2017-01-12 | Nvidia Corporation | Unified optimization method for end-to-end camera image processing for translating a sensor captured image to a display image |
-
2016
- 2016-07-19 CN CN201610567213.1A patent/CN106254720B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170011710A1 (en) * | 2014-01-21 | 2017-01-12 | Nvidia Corporation | Unified optimization method for end-to-end camera image processing for translating a sensor captured image to a display image |
| CN103971354A (en) * | 2014-05-19 | 2014-08-06 | 四川大学 | Method for reconstructing low-resolution infrared image into high-resolution infrared image |
| CN104463793A (en) * | 2014-11-12 | 2015-03-25 | 合肥工业大学 | Video super-resolution reestablishing method and system based on sparse representation and vector continued fraction interpolation under polar coordinates |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107194889A (en) * | 2017-05-11 | 2017-09-22 | 西安电子科技大学 | The bilateral full variational regularization image denoising method of piecemeal |
| CN107194889B (en) * | 2017-05-11 | 2021-02-02 | 西安电子科技大学 | Block bilateral total variation regularization image noise elimination method |
| CN108416371A (en) * | 2018-02-11 | 2018-08-17 | 艾视医疗科技成都有限公司 | A kind of diabetic retinopathy automatic testing method |
| CN108765322A (en) * | 2018-05-16 | 2018-11-06 | 上饶师范学院 | Image de-noising method and device |
| CN108765322B (en) * | 2018-05-16 | 2021-04-27 | 上饶师范学院 | Image denoising method and device |
| CN110208211A (en) * | 2019-07-03 | 2019-09-06 | 南京林业大学 | A kind of near infrared spectrum noise-reduction method for Detecting Pesticide |
| CN110208211B (en) * | 2019-07-03 | 2021-10-22 | 南京林业大学 | Near infrared spectrum noise reduction method for pesticide residue detection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106254720B (en) | 2019-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110443768B (en) | Single-frame image super-resolution reconstruction method based on multiple consistency constraints | |
| Ellmauthaler et al. | Multiscale image fusion using the undecimated wavelet transform with spectral factorization and nonorthogonal filter banks | |
| Zhang et al. | Image compressive sensing recovery via collaborative sparsity | |
| CN103871041B (en) | The image super-resolution reconstructing method built based on cognitive regularization parameter | |
| CN103208102A (en) | Remote sensing image fusion method based on sparse representation | |
| Cheon et al. | Generative adversarial network-based image super-resolution using perceptual content losses | |
| CN105550989B (en) | The image super-resolution method returned based on non local Gaussian process | |
| CN103093433A (en) | Natural image denoising method based on regionalism and dictionary learning | |
| CN106254720A (en) | A kind of video super-resolution method for reconstructing based on associating regularization | |
| CN111145134B (en) | Block effect-based microlens light field camera full-focus image generation algorithm | |
| CN102902961A (en) | Face super-resolution processing method based on K neighbor sparse coding average value constraint | |
| CN102722876A (en) | Residual-based ultra-resolution image reconstruction method | |
| CN110473142A (en) | Single image super resolution ratio reconstruction method based on deep learning | |
| CN103116873B (en) | Image denoising method | |
| CN105513033B (en) | A kind of super resolution ratio reconstruction method that non local joint sparse indicates | |
| CN104504672B (en) | Low-rank sparse neighborhood insertion ultra-resolution method based on NormLV features | |
| CN110211084B (en) | Image multi-resolution reconstruction method based on weight wavelet transform | |
| CN109447903B (en) | Method for establishing semi-reference super-resolution reconstruction image quality evaluation model | |
| CN108280804A (en) | A kind of multi-frame image super-resolution reconstruction method | |
| CN107292316B (en) | Sparse representation-based method for improving image definition | |
| CN104063849A (en) | Video super-resolution reconstruction method based on image block self-adaptive registration | |
| Li et al. | An in-loop filter based on low-complexity CNN using residuals in intra video coding | |
| CN104021523A (en) | Novel method for image super-resolution amplification based on edge classification | |
| CN108460723A (en) | Bilateral full variation image super-resolution rebuilding method based on neighborhood similarity | |
| CN116739899A (en) | Image super-resolution reconstruction method based on SAUGAN network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |