CN102236889A - Super-resolution reconfiguration method based on multiframe motion estimation and merging - Google Patents

Abstract

The invention relates to a super-resolution reconfiguration method based on multiframe motion estimation and merging. In the method, a phase relevant plane is solved from every two adjacent frames in a multiframe image, and multiframe sub-pixel motion estimation is realized on the basis of a parabolic model on the separated two-dimensional phase relevant plane; and a motion vector of a decimal part and a motion vector of an integer part of the sub-pixel motion estimation are applied to a low-resolution image and a high-resolution image respectively, and the current frame image and a plurality of adjacent reference frame images are merged to reconfigure the high-definition image corresponding to the current frame image and provide the reconfigured high-definition image for users, so the ornamental user experience of the users on high-definition video data by network transmission is realized.

Description

A kind of ultra-resolution ratio reconstructing method based on multiframe estimation and fusion

Technical field

The video image that the present invention relates to the communications field is handled, and relates in particular to the ultra-resolution ratio reconstructing method of video image.

Background technology

Along with the fast development of network economy and the continuous maturation of Internet video industry, Internet video market comsupton demand will maintain sustained and rapid growth.Occurred a lot of video website on the internet, these video website operation cost expenditures mainly comprise three bulks: bandwidth, server and popularization, wherein high bandwidth cost allow many medium and small video website can't bear the heavy load.The most cheap also in 1G bandwidth 4,000,000/year, and to make the video smooth playing, realizing viewing and admiring the user experience of level, a video website needs the above bandwidth of 2G at least, also need buy enough servers in various places simultaneously.Under the existing network bandwidth condition, do not support the HD video source, the user terminal video quality definition is not high, has had a strong impact on the popularization of good business model and as the carrying out of IPTV industry.

The resolution of present legal HD video data source is the 1920*1080 pixel, and frame per second was 60 frame/seconds.Because the restriction of the network bandwidth, the spatial resolution that arrives the video data of user side in transmission over networks can only reach 640*480, and frame per second was 12～24 frame/seconds, can't satisfy the user by the view and admire grade user experience of Network Transmission to the HD video data.

Summary of the invention

The technical problem to be solved in the present invention is to utilize the super-resolution reconstruction technology will become the video image of a vertical frame dimension resolution from the multiframe low-resolution video image co-registration that network receives, for example with the low-resolution image of 5 frame 640*480, obtain the high-resolution image of a width of cloth 1920*1080 after the fusion, to satisfy the user under existing bandwidth condition, by the view and admire grade user experience of Network Transmission to the HD video data.

In order to address the above problem, the invention provides a kind of ultra-resolution ratio reconstructing method based on multiframe estimation and fusion, comprise the steps:

(1) utilize phase correlation method to each comprises adjacent described each sub-pix motion vector to two adjacent two field pictures of two two field pictures estimation in all two adjacent two field pictures:

(10) ask the phase place correlation plane of described adjacent two two field pictures;

(12) look for local maximum point on described phase place correlation plane, and according to the ordering of peak value size, extract the bigger one or more peak points of peak value, the position of wherein said one or more peak points is a motion vector;

(14) since actual peak point generally not on integer position, therefore, for in described one or more peak points each, on the bidimensional phase place correlation plane that separates, carry out the sub-pix motion vector estimation: utilize peak point and the pixel that is adjacent to set up parabola model, try to achieve peak point with the corresponding one or more reality of described one or more peak points according to parabola model, the position of the peak point of described reality can be between two integer position, thereby obtain the sub-pix motion vector of two adjacent two field pictures;

(2) based on each sub-pix motion vector of being estimated to two adjacent two field pictures, current frame image and a plurality of reference frame image that are adjacent are merged the HD image that becomes the given resolution corresponding with current frame image, specifically comprise:

(20) current frame image is amplified to the high-definition picture of described given resolution, as the initial value of iteration;

(22) be that unit carries out following step with the image block to each frame in the reference frame image:

(2210) the sub-pix motion vector of the image block correspondence of reference frame is resolved into integral part and fraction part, the image block of described reference frame is moved the motion vector of described integral part, the image block of the reference frame that obtains moving, the image block of described high-definition picture that will be corresponding with the image block of reference frame moves the motion vector of described fraction part, the image block of the high-definition picture that obtains moving;

(2220) image block to described mobile high-definition picture carries out gaussian filtering and down-sampling, to obtain the down-sampled images piece identical with the resolution of described reference frame image;

(2230) image block of described down-sampled images piece and described mobile reference frame is subtracted each other obtain subtracting image block;

(2240) the described image block that subtracts is carried out up-sampling and Gauss's low-pass filtering to obtain the up-sampling image block of described given resolution;

(2250) with the reverse motion vector that moves described fraction part of described up-sampling image block, with the up-sampling image block that obtains oppositely moving;

(2260) the described up-sampling image block that oppositely moves be multiply by iteration step length β and output, the scope of the value of wherein said iteration step length β between 0 to 1;

(24) the output result with the described step (2260) of all reference frame image is added to described high-definition picture, to upgrade described high-definition picture;

(26) return described step (22) iteration to obtain the HD image of described given resolution, iterations rule of thumb is worth definite.

Method of the present invention, in described step (14) afterwards, also comprise step: (16) are according to the peak value of the peak point of described one or more reality, calculate the relative peak of the peak point of described one or more reality, described relative peak is represented divided by the peak value sum of all true peak points with the peak value of each actual peak point;

In described step (2230) afterwards, also comprise step: (2235) ask the described absolute value sum that subtracts image block, and obtain gain from gain model according to described absolute value sum, described gain be multiply by the relative peak of motion vector correspondence of the image block of described reference frame, to obtain dependability parameter, wherein said gain model is the function of described absolute value sum, and time gain is 1 more than or equal to predetermined threshold when described absolute value sum, when described absolute value sum during less than predetermined threshold gain linearity descend;

In described step (2250) afterwards, also comprise step: (2255) multiply by described dependability parameter to revise the described up-sampling image block that oppositely moves with the described up-sampling image block that oppositely moves.

Method of the present invention, wherein said step (10) also comprises step:

(1010) use window function that described two adjacent two field pictures are carried out windowing process respectively;

(1020) two two field pictures to windowing process carry out fast fourier transform;

(1030) plural form that the real part of Fourier transform results is added imaginary part converts the plural form of amplitude and phase place to;

(1040) the phase place correspondence of the Fourier transform results of two two field pictures is subtracted each other, obtain phase differential;

(1050) described phase differential being converted to amplitude is 1 the amplitude and the plural form of phase place;

(1060) carrying out fast to the transformation result of described step (1050), inverse-Fourier transform obtains the phase place correlation plane.

Method of the present invention, in the wherein said step (1010), described window function is a Hanning window.

Method of the present invention, wherein said step (1010) also comprises before: described two adjacent two field pictures are carried out pre-service respectively, and described pre-service comprises denoising or down-sampling.

Method of the present invention, the plural form that converts complex value and phase place in wherein said step (1030) and the described step (1020) to utilizes the CORDIC cordic algorithm to realize.

The present invention asks for the phase place correlation plane by each adjacent two frame from multiple image, and on the bidimensional phase place correlation plane that separates, realize multiframe sub-pix estimation based on parabola model, obtain in the frame accurately, reliably and the movable information of the corresponding same object of interframe; And then by the motion vector of the fraction part of this sub-pix estimation and the motion vector of integral part are applied to low-resolution image and high-definition picture respectively, current frame image and adjacent a plurality of reference frame image are merged, the HD image corresponding with current frame image with reconstruct offers the user, thereby realized that the user is by the view and admire grade user experience of Network Transmission to the HD video data.

Description of drawings

Fig. 1 is the selection synoptic diagram of correlation plane calculation window size of the present invention;

Fig. 2 is the synoptic diagram of phase place correlation plane of the present invention in the image internal separation;

Fig. 3 is the block diagram of the relevant estimation of phase place of the present invention;

Fig. 4 is the peak Distribution synoptic diagram of phase place correlation plane;

Fig. 5 is a synoptic diagram of asking for the sub-pixel location of true peak correspondence under parabola model;

Fig. 6 is the synoptic diagram that true picture is degenerated to the process of low-resolution image;

Fig. 7 be among the present invention the object piece in the frame of video corresponding to the synoptic diagram of multiple image;

Fig. 8 is frame of video F of the present invention _N-1The synoptic diagram of fusion process;

Fig. 9 is the gain model of the absolute value sum of the difference based on image block of the present invention;

Figure 10 is the synoptic diagram of multiframe iteration fusion process of the present invention;

Figure 11 is a system of the present invention general diagram.

Embodiment

Be described in further detail below in conjunction with the embodiment of accompanying drawing technical scheme of the present invention:

The enforcement of technical scheme proposed by the invention is based on two hypothesis:

1. in frame, there are some similar structure or details.

2. in interframe, because the sampling phase difference, image exists the different information of same object.

Simultaneously based on two processes:

1. obtain in the frame with super sub-pix estimation and the information of the same object of interframe correspondence.

2. self-adaptation merges these all information.

Phase place correlation plane estimation principle:

Infinite precision moving displacement between phase place correlation plane (PPC) energy reflecting video.Its ultimate principle is:

Suppose the motion that a translation is arranged at two width of cloth images:

F _n(n1，n2)＝F _n-1(n1+d1，n2+d2) (1)

Behind the two-dimension fourier transform:

S _n(f ₁，f ₂)＝S _n-1(f ₁，f ₂)exp[j2π(d1f1+d2f2)] (2)

The Fourier transform of two frame simple crosscorrelation is:

C _{N, n-1}(f1, f2)=S _N-1(f1, f2) S _n(f1, f2) (3) in order to eliminate the influence that brightness changes, the normalization spectrum is:

$\mathrm{\&Phi;}\left[C_{n,n-1}(\mathrm{<figure-callout\; id="1"\; label="f"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">f</figure-callout>}1,f2)\right]=\frac{S_{n-1}(\mathrm{<figure-callout\; id="1"\; label="f"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">f</figure-callout>}1,f2)S_n^{*}(\mathrm{<figure-callout\; id="1"\; label="f"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">f</figure-callout>}1,f2)}{\left|S_{n-1}(\mathrm{<figure-callout\; id="1"\; label="f"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">f</figure-callout>}1,f2)S_n^{*}(\mathrm{<figure-callout\; id="1"\; label="f"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">f</figure-callout>}1,f2)\right|}=\exp [-\mathrm{<figure-callout\; id="2"\; label="j"\; filenames="DEST\_PATH\_HSB00000289780900041.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">j</figure-callout>}2\mathrm{\&pi;}(f1\mathrm{<figure-callout\; id="1"\; label="d"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">d</figure-callout>}1+f2d2)]---\left(4\right)$

The following formula two-dimension inverse transformation is:

c _n，n-1(n1，n2)＝δ(n1-d1，n2-d2) (5)

By following formula (5) as can be known, the peak on the phase place correlation plane is corresponding to the motion vector between two width of cloth images.

When asking for the phase place correlation plane, taken into full account and assessed the cost and the accuracy of estimation, and the realization of fast fourier transform, the data block of having chosen the window of long by 128, wide 64 pixels is that unit asks for the phase place correlation plane.Consider edge effect, this data block is only effective to 64 * 32 windows of center, as shown in Figure 1.

With reference to figure 2, in video image, the plane of delineation is divided into nonoverlapping 64 * 32 image block, uses 128 * 64 overlapping image blocks to ask for the phase place correlation plane then, thereby ask for a plurality of sub-pix motion vectors of 64 * 32 image blocks of the center of this 128 * 64 image block.In follow-up fusion process, this 64 * 32 image block further is divided into 32 8 * 8 image block, operate with the image block of 8 * 8 windows during fusion.

Be phase place correlation plane estimation implementation procedure below:

Block diagram shown in Figure 3 is a phase place correlation plane of asking for 128 * 64 image blocks of two frame correspondences, and the process of obtaining the output of the sub-pix motion vector of this piece and corresponding peak value according to the phase place correlation plane.Processing procedure comprises:

Pre-service: mainly be at the big video of noise, carry out denoising.To assess the cost in order reducing in addition, can to adopt down-sampling to reduce the resolution of processing.

Windowing process: Fourier is that signal is the cycle to the prerequisite supposition of signal Processing, because the most of situation in 128 * 64 image block left sides and the right is not same object, the people is the existence that has caused high-frequency information, has promptly influenced accurately obtaining of final vector.Therefore with window function this image block is carried out windowing process.Preferably, use Hanning window Hanning to be added on this image block.

Convert plural form to the amplitude phase place:, obtain plural number: x+yj through behind the two-dimension fourier transform; Ask for phase differential for convenience, plural form need be expressed as: amplitude is

Phase place is Form.When computer software and hardware was realized, above-mentioned amplitude and phase place need be approached by iteration, preferably, utilize the CORDIC cordic algorithm to realize the plural form of real part imaginary part is converted to the form of amplitude phase place.

Phase differential: the phase place correspondence that the correspondence image piece of adjacent two frames is tried to achieve is subtracted each other.

Convert phase differential to plural form: utilizing the CORDIC cordic algorithm that phase differential is converted to amplitude is 1 plural form, carries out inverse fourier transform then.

Local peaking is selected: Figure 4 shows that correlation plane, find the peak value of local maximum from correlation plane, and therefrom select peak value bigger preceding 2 or 4 or more a plurality of peak value.Correlation plane data block center is (0,0), and the peak value present position is the value of motion vector.

Near sub-pix estimation:, as shown in Figure 5, suppose and satisfy parabola model the peak value on the correlation plane because the integer position of the peak value of phase place correlation plane may not be actual peak value position:

f(y)＝ay ²+by+c

$\mathrm{dy}=\frac{c_{n,n-1}(\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">n</figure-callout>}1+1,n2)-c_{n,n-1}(n1-1,n2)}{2({2c}_{n,n-1}(\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">n</figure-callout>}1,n2)-c_{n,n-1}(\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="DEST\_PATH\_HSB00000289780900051.png,DEST\_PATH\_HSB00000354043800021.png"\; state="\{\{state\}\}">n</figure-callout>}1+1,n2)-c_{n,n-1}(n1-1,n2))}---\left(6\right)$

According to parabola model, the two-dimensional data of separating on the employing phase place correlation plane is asked for the peak point with the corresponding reality of peak point respectively, the position of the peak point of the reality of being asked for can be between two integer position, thereby obtain the sub-pix motion vector of two adjacent two field picture pieces.

Relative peak calculates: the relative peak that calculates actual peak point according to the peak value of the peak point of reality, described relative peak represents that divided by the peak value sum of all true peak points the size of relative peak has been reacted the reliability of corresponding motion vector with the peak value of each actual peak point.In order to encourage unimodal value, with 2 peak value peak0, peak1 is that example is calculated relative peak:

Peak0＝peak0/(peak0+peak1)

Peak1＝peak1/(peak0+peak1) (7)

The super-resolution multiframe merges

At first consider the model of image degradation, as shown in Figure 6:

Note: X is real natural scene, The atmosphere fuzzy effect;

T _kObject of which movement, Camera is fuzzy;

D _kDown-sampling, V _kNoise;

F _kThe low-resolution video that observes.

The expression formula of the degenerative process of Fig. 6 is:

$F_k=D_k{H}_k^{\mathrm{cam}}{T}_k{H}_k^{\mathrm{atm}}X+V_k---\left(8\right)$

Motion does not generally influence the position of atmosphere fuzzy operator, therefore has:

$=D_k{H}_k^{\mathrm{cam}}{H}_k^{\mathrm{atm}}{T}_{k}X+V_k,k=1,\&CenterDot;\&CenterDot;\&CenterDot;,N$

By following energy function minimum, ask X:

$\hat{X}=\underset{X}{\mathrm{ArgMin}}\left[\underset{k=1}{\overset{N}{\mathrm{\&Sigma;}}}{\left|\right|D_k{H}_k{T}_{k}X-F_k\left|\right|}_2^2\right]$

Steepest gradient solution by iterative method:

${\hat{X}}_{t+1}={\hat{X}}_t-\mathrm{\&beta;}\left\{\underset{k=1}{\overset{N}{\mathrm{\&Sigma;}}}{T}_k^T{H}_k^T{D}_k^T(D_k{H}_k{T}_k{\hat{X}}_t-F_k)\right\}---\left(10\right)$

Super-resolution Fusion Model and process:

The sub-pix estimation that is based on shown in Figure 7, reference frame image piece correspondence the synoptic diagram of multiframe information.Referring to Fig. 7, any F _nLast 8 * 8 image blocks, a plurality of image blocks on corresponding the reference frame: M image block for example, depend on that to choose several motion vectors for a phase place correlation plane relevant.The contribution of each piece simultaneously is different, is designated as α _Kl, following formula can be written as:

${\hat{X}}_{t+1}={\hat{X}}_t-\mathrm{\&beta;}\left\{\underset{k=1}{\mathrm{\&Sigma;}}\underset{l=1}{\mathrm{\&Sigma;}}{\mathrm{\&alpha;}}_{\mathrm{kl}}{T}_k^T\left(l\right)H_k^T{D}_k^T(D_k{H}_k{T}_k\left(l\right){\hat{X}}_t-F_k)\right\}---\left(11\right)$

Based drive relativity, and motion vector is a sub-pixel precision.Therefore, high-definition picture is moved the fraction part of sub-pix motion vector, and low-resolution image is moved the whole branch of sub-pix motion vector.Avoided bilinear interpolation like this.

${\hat{X}}_{t+1}={\hat{X}}_t-\mathrm{\&beta;}\left\{\underset{k=n-2}{\mathrm{\&Sigma;}}\underset{l=1}{\mathrm{\&Sigma;}}{\mathrm{\&alpha;}}_{\mathrm{kl}}{\mathrm{Tfrac}}_k^T\left(l\right)H_k^T{D}_k^T(D_k{H}_k{\mathrm{Tfrac}}_k\left(l\right){\hat{X}}_t-{\mathrm{TInt}}_k\left(l\right)F_k)\right\}---\left(12\right)$

Wherein, Tfrac _k(l) be T _k(l) fraction part, TInt _k(l) be T _k(l) integral part

Block representation shown in Figure 8 fusion video image F in the following formula _N-1Process, be unit with low resolution 8 * 8 image blocks during processing.Among the figure:

: be current image block sub-pix motion vector;

: the relative peak of motion vector institute corresponding phase correlation plane, see formula (7); Calculation of Reliability: an input of Calculation of Reliability module is

, another input is image block poor of the image block of mobile fraction part and down-sampling and mobile integral part.

Value represent the reliability of this motion vector, the difference of this image block is represented the coupling reliability of 8 * 8 data to a certain extent.Be the gain model of setting up as shown in Figure 9 based on the absolute value sum of the difference of image block, when the absolute value sum is less than or equal to threshold value Th, gain G=1; When absolute value sum during greater than threshold value Th, gain linearity descends.At this moment, data do not match.Final Calculation of Reliability is as follows:

${\mathrm{\&alpha;}}_{n-1,l}={\mathrm{Flag}}_{F_n\&LeftRightArrow;F_{n-1}}\&times;G;---\left(13\right)$

Figure 10 has reflected Fusion Model formula (12); Figure 10 only shows the fusion of 5 two field pictures, can be generalized to any frame number and merges.

Figure 11 is whole super-resolution algorithms general diagram.Can be embodied as hardware, also can be embodied as software based on PC and DSP.

Above-mentioned to the having been described in detail of technical scheme of the present invention, the preferred embodiment of the ultra-resolution ratio reconstructing method based on multiframe sub-pix estimation and fusion of the present invention comprises the steps:

The first step: utilize phase correlation method to each comprises the adjacent corresponding sub-pix motion vector of two two field pictures estimation in all two adjacent two field pictures:

(101) from internal memory, read two adjacent two field pictures;

(102) two adjacent two field pictures are carried out pre-service respectively, pre-service comprises denoising or down-sampling;

(103) use window function that two adjacent two field pictures are carried out windowing process respectively, avoid the extra high frequency owing to the data truncation generation, wherein window function can be Hanning window (Hamming), Kaiser window or quarter window;

(104) fast two-dimensional fourier transformation;

(105) plural form that utilizes cordic algorithm that the real part of Fourier transform results is added imaginary part converts the form of amplitude and phase place to;

(106) the phase place correspondence of the Fourier transform results of two two field pictures is subtracted each other, obtain phase differential;

(107) phase differential being converted to amplitude is the plural form that 1 real part adds imaginary part;

(108) carrying out fast to the transformation result of step (17), inverse-Fourier transform obtains the phase place correlation plane.

(109) look for local maximum point on the phase place correlation plane, and according to the ordering of peak value size, extract the bigger one or more peak points of peak value, the position of wherein said one or more peak points is a motion vector;

(110) since actual peak point generally not on integer position, therefore, for in described one or more peak points each, on the bidimensional phase place correlation plane that separates, carry out the sub-pix motion vector estimation: utilize peak point and the pixel that is adjacent to set up parabola model, try to achieve the peak point of one or more reality according to parabola model, the position of the peak point of described reality can be between two integer position, thereby obtain the sub-pix motion vector of two adjacent two field pictures;

(111) relative peak of the peak point of the one or more reality of calculating, this relative peak is represented divided by the peak value sum of all true peak points with the peak value of each actual peak point;

(112) the sub-pix motion vector and the pairing relative peak that will obtain between adjacent two frames is saved in the internal memory.

Second step: based on each sub-pix motion vector of being estimated, current frame image and a plurality of reference frame image that are adjacent are merged to rebuild corresponding with current frame image HD image, specifically comprise to two adjacent two field pictures:

(210) read the low resolution present frame, arrive desired high-definition picture with bicubic interpolation method enlarged image, as the initial value of iteration;

(220), be that unit carries out following step with the image block to each frame in the reference frame image in order to merge reference frame image:

(2210) the sub-pix motion vector of the image block correspondence of reference frame is resolved into integer portion

Divide and fraction part, the image block of reference frame is moved the motion vector of integral part, moved

The image block of moving reference frame moves the motion vector of fraction part, the high-definition picture piece that obtains moving with the high-definition picture piece;

(2220) the high-definition picture piece that is moved is carried out gaussian filtering and down-sampling, to obtain the down-sampled images piece identical with the resolution of described reference frame image;

(2230) image block of down-sampled images piece and the reference frame that moved is subtracted each other obtain subtracting image block;

(2235) ask the absolute value sum that subtracts image block, and obtain gain from gain model according to described absolute value sum, gain be multiply by the relative peak of motion vector correspondence of the image block of reference frame, to obtain dependability parameter, wherein gain model is the function of absolute value sum, and time gain is 1 more than or equal to predetermined threshold when the absolute value sum, when the absolute value sum during less than predetermined threshold gain linearity descend;

(2240) image block be will subtract and up-sampling and Gauss's low-pass filtering carried out to obtain the up-sampling image block of given resolution;

(2250) with the motion vector of the reverse mobile fraction part of up-sampling image block, with the up-sampling image block that obtains oppositely moving;

(2255) the up-sampling image block that will oppositely move multiply by dependability parameter to revise the up-sampling image block that oppositely moves.

(2260) the up-sampling image block that will oppositely move multiply by iteration step length β and output, the wherein scope of the value of iteration step length β between 0 to 1;

(240) the output result with the step (2260) of all reference frame image is added to high-definition picture, to upgrade high-definition picture;

(260) return described step (220) iteration to obtain the HD image of described given resolution, iterations rule of thumb is worth definite.

Certainly; the present invention also can have other various embodiments; under the situation that does not deviate from spirit of the present invention and essence thereof; those of ordinary skill in the art work as can make various corresponding changes and distortion according to the present invention, but these corresponding changes and distortion all should belong to the protection domain of the appended claim of the present invention.

Claims (7)

1. one kind based on the estimation of multiple image and the ultra-resolution ratio reconstructing method of fusion, comprises the steps:

(1) utilize phase correlation method to each comprises adjacent described each sub-pix motion vector to two adjacent two field pictures of two two field pictures estimation in all two adjacent two field pictures:

(10) ask the phase place correlation plane of described adjacent two two field pictures;

(12) look for local maximum point on described phase place correlation plane, and according to the ordering of peak value size, extract the bigger one or more peak points of peak value, the position of wherein said one or more peak points is a motion vector;

(14) since actual peak point generally not on integer position, therefore, for in described one or more peak points each, on the bidimensional phase place correlation plane that separates, carry out the sub-pix motion vector estimation: utilize peak point and the pixel that is adjacent to set up parabola model, try to achieve peak point with the corresponding one or more reality of described one or more peak points according to parabola model, the position of the peak point of described reality can be between two integer position, thereby obtain the sub-pix motion vector of two adjacent two field pictures;

(2) based on each sub-pix motion vector of being estimated to two adjacent two field pictures, current frame image and a plurality of reference frame image that are adjacent are merged the HD image that becomes the given resolution corresponding with current frame image, specifically comprise:

(20) current frame image is amplified to the high-definition picture of described given resolution, as the initial value of iteration;

(22) be that unit carries out following step with the image block to each frame in the reference frame image:

(2210) the sub-pix motion vector of the image block correspondence of reference frame is resolved into integral part and fraction part, the image block of described reference frame is moved the motion vector of described integral part, the image block of the reference frame that obtains moving, the image block of described high-definition picture that will be corresponding with the image block of reference frame moves the motion vector of described fraction part, the image block of the high-definition picture that obtains moving;

(2220) image block to described mobile high-definition picture carries out gaussian filtering and down-sampling, to obtain the down-sampled images piece identical with the resolution of described reference frame image;

(2230) image block of described down-sampled images piece and described mobile reference frame is subtracted each other obtain subtracting image block;

(2240) the described image block that subtracts is carried out up-sampling and Gauss's low-pass filtering to obtain the up-sampling image block of described given resolution;

(2250) with the reverse motion vector that moves described fraction part of described up-sampling image block, with the up-sampling image block that obtains oppositely moving;

(2260) the described up-sampling image block that oppositely moves be multiply by iteration step length β and output, the scope of the value of wherein said iteration step length β between 0 to 1;

(24) the output result with the described step (2260) of all reference frame image is added to described high-definition picture, to upgrade described high-definition picture;

(26) return described step (22) iteration to obtain the HD image of described given resolution, iterations rule of thumb is worth definite.

2. the method for claim 1, in described step (14) afterwards, also comprise step: (16) are according to the peak value of the peak point of described one or more reality, calculate the relative peak of the peak point of described one or more reality, described relative peak is represented divided by the peak value sum of all true peak points with the peak value of each actual peak point;

In described step (2230) afterwards, also comprise step: (2235) ask the described absolute value sum that subtracts image block, and obtain gain from gain model according to described absolute value sum, described gain be multiply by the relative peak of motion vector correspondence of the image block of described reference frame, to obtain dependability parameter, wherein said gain model is the function of described absolute value sum, and time gain is 1 more than or equal to predetermined threshold when described absolute value sum, when described absolute value sum during less than predetermined threshold gain linearity descend;

In described step (2250) afterwards, also comprise step: (2255) multiply by described dependability parameter to revise the described up-sampling image block that oppositely moves with the described up-sampling image block that oppositely moves.

3. method as claimed in claim 1 or 2, wherein said step (10) also comprises step:

(1010) use window function that described two adjacent two field pictures are carried out windowing process respectively;

(1020) two two field pictures to windowing process carry out fast fourier transform;

(1030) plural form that the real part of Fourier transform results is added imaginary part converts the form of amplitude and phase place to;

(1040) the phase place correspondence of the Fourier transform results of two two field pictures is subtracted each other, obtain phase differential;

(1050) described phase differential being converted to amplitude is 1 the amplitude and the plural form of phase place;

(1060) carrying out fast to the transformation result of described step (1050), inverse-Fourier transform obtains the phase place correlation plane.

4. method as claimed in claim 3, in the wherein said step (1010), described window function is a Hanning window.

5. method as claimed in claim 4, wherein said step (1010) also comprises before: described two adjacent two field pictures are carried out pre-service respectively, and described pre-service comprises denoising or down-sampling.

6. method as claimed in claim 4, the plural form that converts complex value and phase place in wherein said step (1030) and the described step (1020) to utilizes the CORDIC cordic algorithm to realize.

7. method as claimed in claim 5, the plural form that converts complex value and phase place in wherein said step (1030) and the described step (1020) to utilizes the CORDIC cordic algorithm to realize.

Images