CN105100807A - Motion vector post-processing based frame rate up-conversion method - Google Patents

Abstract

The invention provides a motion vector post-processing based frame rate up-conversion method. According to the method, firstly, forward and backward motion estimations are performed on an original video frame, so as to obtain a forward motion vector and a backward motion vector, and abnormal values of the forward motion vector and the backward motion vector are modified based on prediction; bidirectional candidate motion vectors are selected based on minimum sum of absolute differences (SAD); then, different smoothing methods are adopted for bidirectional motion vectors of different positions; vector extrapolation is performed on an edge block and four corner blocks firstly, a distance-based weighting smoothing method is adopted for unreliable motion vectors , the candidate vectors of a middle block are selected based on the direction and magnitude of the motion vector of an adjacent block, and a final vector of a to-be-interposed block is selected from the candidate vectors based on the minimum SAD; and finally, an interpolation frame is obtained by use of bidirectional motion compensation of overlapped blocks. Through adoption of the frame rate up-conversion method, the problems of holes and shielding of the interpolation frame can be solved effectively, phenomena of motion blur and shaking can be eliminated, and the visual effect of the constructed interpolation frame is improved.

Description

A kind of frame per second method for improving based on motion vector reprocessing

Technical field

The present invention relates to a kind of method that video frame rate promotes, belong to field of video image processing.

Background technology

Frame per second lift technique a kind ofly utilizes the time and space correlation of consecutive frame in original video frame sequence to generate new frame of video and be inserted in original video frame to promote the technology of former video frame rate.Video frame rate is higher, and the picture play in the unit interval is more, then flickering when picture replaces is more weak, and the picture of reproduction is more level and smooth, also finer and smoother.High frame per second can obtain animation more smooth, more true to nature, and each second, frame number was more, and shown action will be more smooth, and the texture of picture is also better.Therefore, find a kind of frame per second lift technique to be necessary.

Frame per second lift technique can be divided into two classes: one simply utilizes frame repetition or frame on average to obtain interpolated frame, but this method can not provide gratifying visual experience, and interpolated frame can be made to occur shake, motion blur phenomenon or ghost effect.Another kind is the motion compensation frame interpolation method based on movable information, is frame per second lift technique the most frequently used at present, mainly comprises estimation, motion vector reprocessing and motion compensation three steps.And the key point of this method be exactly motion estimation stage obtain the accuracy of motion vector, motion vector is more accurate, more can reflect the real motion of former video, thus the interpolation frame obtained is more accurate.But at present conventional block-based method for estimating often produce block, cavity and overlap problem and cause interpolation frame to there is the problem such as blocking effect and motion blur due to the inaccuracy of motion vector.

Summary of the invention

In order to obtain more accurate motion vector, solve during frame per second promotes the interpolation frame blocking effect and motion blur problems that cause due to motion vector inaccuracy, the present invention proposes a kind of new motion vector reprocessing scheme, and edge blocks, four hornblocks and intermediate mass adopt different smoothing methods respectively.That is: the advanced row vector extrapolation of edge blocks and four hornblocks, and then the weighted motion vector carried out based on distance the edge block after extrapolation is level and smooth; Motion vector candidates based on the direction of motion is first carried out to intermediate mass, then choosing based on least absolute error and (SAD, SumofAbsoluteDifferences) is adopted to candidate vector; In order to solve in estimation easily occur block, cavity and overlap problem, the present invention adopts a kind of method for estimating combined based on three-dimensional, combines by forward, backward and bi-directional motion estimation; In order to make motion vector more accurate, the adaptive motion vector correction based on prediction is carried out to after forward direction vector to vector respectively after forward, backward estimation, reject the exceptional value in vector, and then choose optimum orientation vector as bi-directional motion vector.

In order to realize foregoing invention object, the present invention adopts following technical proposals:

Based on a frame per second method for improving for motion vector reprocessing, it is characterized in that: at motion estimation stage, no longer carry out unidirectional estimation, but have employed the motion estimation scheme that three different directions combine; In motion vector post-processing stages, no longer adopt unified medium filtering, but adopt different self adaptation vector corrections and Smooth scheme for the motion vector of different directions, diverse location, concrete steps are:

Step 1: process original video, is treated to frame;

Step 2: the forward and backward estimation based on diamond block search is carried out respectively to two continuous frames;

Step 3: to estimating that the forward and backward motion vector obtained carries out the adaptive motion vector correction based on prediction respectively;

Step 4: select the vector of optimum orientation as two-way candidate motion vector in revised forward and backward motion vector;

Step 5: location-based adaptive motion vector carries out to two-way candidate motion vector level and smooth;

Step 6: two-way overlapped block motion compensation obtains inserting frame;

Step 7: video insertion frame and primitive frame being synthesized high frame per second.

Preferably, first rejecting outliers is carried out to forward and backward motion vector in step 3, then the adaptive motion vector correction based on prediction is adopted to exceptional value block vector, to reject vector exceptional value, improve the accuracy of one-way movement vector.

Preferably, in step 4 respectively based on absolute error and the sad value of forward and backward motion vector computation match block and current block, choosing the little motion vector of corresponding sad value is two-way candidate motion vector, thus revises the inaccurate motion vector because occlusion issue causes.

Preferably, in steps of 5 according to treating inserted block diverse location, carry out adaptive motion vector smoothly to two-way candidate motion vector, concrete grammar is: edge blocks and the advanced row vector extrapolation of four hornblock vectors, then adopts the weight vectors based on distance level and smooth; Horizontal direction and the judgement of vertical direction scope are first carried out to intermediate mass, to determine the candidate vector treating inserted block, the sad value treating current block in match block and present frame in the search frame that inserted block is corresponding is calculated respectively again based on candidate vector, choosing the minimum motion vector of corresponding sad value is optimal motion vectors, revises inaccurate motion vector.

The method efficiently solves cavity in interpolation frame and occlusion issue, eliminates the motion blur because motion vector inaccuracy causes and jitter phenomenon, improves the visual effect that structure inserts frame.

Accompanying drawing explanation

Fig. 1 is flow chart of the present invention;

Fig. 2 is forward, backward estimation schematic diagram;

Fig. 3 is current block and surrounding block motion vector schematic diagram thereof;

Fig. 4 is comparison diagram before and after the correction of one-way movement vector;

Fig. 5 is motion vectors mapping figure and treats that interleave intermediate mass candidate vector selects key diagram;

Fig. 6 treats interleave corner block and edge block schematic diagram and vector extrapolation figure thereof;

Fig. 7 is two-way overlapped block motion compensation schematic diagram;

Fig. 8 is Foreman video simulation Comparative result figure.

Specific embodiments

The present invention proposes a kind of frame per second method for improving based on motion vector reprocessing.Forward, backward and the two-way method for estimating combined are carried out respectively to former video two continuous frames, the motion vector of different directions is adopted to the self adaptation reprocessing of different schemes, recycling optimum movement vector carries out overlapping block bi directional motion compensation, and then reach and solve cavity in interpolation frame and occlusion issue and eliminate motion blur and jitter phenomenon, improve the object that structure inserts the visual effect of frame.

Clearly illustrate the overall flow of the method for the invention in Fig. 1, concrete implementation step is as follows:

(1) read in video, arrange counter t=1, preserve t frame successively as present frame, t+2 frame, as with reference to frame, reserves t+1 frame as incoming frame to be inserted;

(2) one-way movement is estimated: carry out the forward and backward estimation based on DS Block-matching to present frame and reference frame, as shown in Figure 2, obtains forward motion vector v _forwardwith backward motion vector v _backward;

(3) one-way movement vector correction: respectively to v _forwardand v _backwardcarry out the adaptive correction based on prediction, with v _forwardfor example, it is as follows that it revises step:

A. motion vector rejecting outliers:

Step1: calculate decision block (being designated as B block) motion vector v ₁and the motion vector (v of surrounding eight blocks ₂-v ₉, mean value v as shown in Figure 3) _m, as shown in Equation 1.

$v_m=\frac{1}{9}\underset{i=1}{\overset{9}{\Σ}}{v}_i$ (formula 1)

Step2: calculate v ₁and v _mthe absolute difference of value:

D _c=| v _m-v ₁| (formula 2)

Step3: calculate each adjacent block motion vector value and v _mthe mean value of absolute difference:

$D_n=\frac{1}{8}\underset{i=2}{\overset{9}{\Σ}}|v_m-v_i|$ (formula 3)

Step4: judge, if D _c> D _n, then v ₁for motion vector exceptional value, need v ₁revise.

B. the correction based on prediction is carried out to motion vector exceptional value:

Step1: the motion vectors v calculating exceptional value block B block _pre:

V _pre=(v ₂+ v ₃+ v ₄+ v ₅)/4 (formula 4)

Step2: calculate eight neighboring block motion vector (v respectively ₂-v ₉) and v _prethe absolute difference of value, choose 4 motion vectors that absolute difference in 8 vectors is minimum, average to it, gained is amended current block motion vector.

Accompanying drawing 4 illustrates the motion vector field before and after motion vector refinement, and in (a), dotted line vector is exceptional value, and in (b), dotted line vector is the vector after correcting.

(4) the choosing of two-way candidate motion vector: respectively based on the sad value corresponding to each block of forward and backward motion vector computation: SAD _forwardand SAD _backward, the half of the vector corresponding to sad value relatively little in both is two-way candidate motion vector v _bi.As shown in Equation 5:

${\mathrm{SAD}}_{forward}(B_{i,j},v_{forward})=\underset{s\∈B_{i,j}}{\Σ}|f_t\left(s\right)-f_{t+2}(s+v_{forward})|$

${\mathrm{SAD}}_{backward}(B_{i,j},v_{backward})=\underset{s\∈B_{i,j}}{\Σ}|f_{t+2}\left(s\right)-f_t(s+v_{backward})|$

$v_{bi}=\left\{\begin{array}{c}\frac{v_{forward}}{2},if:SA{D}_{forward}\≤SA{D}_{backward}\\ -\frac{v_{backward}}{2},if:{\mathrm{SAD}}_{forward}>{\mathrm{SAD}}_{backward}\end{array}\right.$ (formula 5)

Note: SAD in formula _forwardand SAD _backwardrepresent respectively absolute error on forward and backward direction and, f _t(s) and f _t+2s () represents the pixel value at coordinate position s place in present frame and reference frame respectively, B _i,jrepresent the scope of block.

(5) two-way candidate motion vector is level and smooth: during by step (4) gained optimum orientation motion vectors mapping to interpolation frame, the interior inserted block A that the block B placed is general and corresponding also not exclusively overlaps, as shown in accompanying drawing 5 (a), so just causing placing block vector can not the real motion of complete reaction object, therefore in order to improve the accuracy of vector, smoothing to interpolation frame intermediate mass, edge block and corner block motion vector respectively.

A. intermediate mass smoothed motion vector:

When the intermediate mass mapping movements vector to interpolation frame, according to direction and the size of object of which movement, place block B in a certain direction with treat that inserted block A may have the shown three kinds of situations of Fig. 5 (b).Therefore can judge that the block of inserted block overlap is treated in eight neighborhood block neutralizations according to the direction and number range for the treatment of inserted block surrounding neighbors block motion vector, get the vector of the neighborhood block that overlaps as the candidate vector treating inserted block, and then treat inserted block with candidate vector respectively and carry out SAD calculating, get make sad value minimum candidate vector as the final vector treating inserted block, as shown in Equation 6.If treat, inserted block does not have candidate vector, then the motion vector medium filtering treating eight blocks around inserted block obtains final vector, as shown in Equation 7.

$v_{final\_bi}=\arg \underset{v_{bi}\∈V}{min}\left\{\underset{s\∈B_{i,j}}{\mathrm{\Σ}}|f_t\left(s-v_{bi}\right)-f_{t+2}\left(s+v_{bi}\right)|\right\}$ (formula 6)

V _{final_bi}=median{v ₂, v ₃, v ₄, v ₅, v ₆, v ₇, v ₈, v ₉(formula 7)

Note: B in formula 6 _i,jrepresent the scope of block, behalf treats the position coordinates of inserted block, and the candidate set of inserted block is treated in V representative; In formula 7, v ₂-v ₉namely the motion vector of the neighborhood block of inserted block is treated

The direction that accompanying drawing 5 (c) illustrates neighborhood block vector as indicated with an arrow to time, the neighborhood block of placement with treat that inserted block A can overlap.

B. the smoothed motion vector of edge block and four hornblocks:

Before the smoothed motion vector of corner to be inserted block (Fig. 6 (a)) and edge block (Fig. 6 (b)), first treat inserted block based on neighborhood block and carry out Motion vector extrapolation, form 3*3 neighborhood block, treat inserted block B with the upper left corner ₁with left side edge block B ₂for example, its Motion vector extrapolation figure is as shown in accompanying drawing 6 (c) He Fig. 6 (d), and wherein dashed boxes is outer inserted block.Then based on Motion vector extrapolation figure, the motion vector treating inserted block carries out reliability judgement, adopts level and smooth based on the weight vectors of distance to insecure motion vector.Inserted block B is treated with the upper left corner ₁for example, concrete smoothing step is:

Step1: edge treats inserted block B ₁motion vector extrapolation, obtains Fig. 6 (c), arranges to obtain Fig. 6 (e) to motion vector order, and following steps all operate for Fig. 6 (e);

Step2: judge that inserted block B is treated at edge ₁motion vector v ₀reliability: v in computing formula 8 _m, D _nand D _cnumerical value:

$v_m=\frac{1}{9}\underset{i=0}{\overset{8}{\Σ}}-v_m$

$D_n=\frac{1}{9}\underset{i=0}{\overset{8}{\Σ}}|v_i-v_m|$ (formula 8)

D _c＝|v ₀-v _m|

Note: v in above formula _mrepresent and treat inserted block and eight neighborhood block motion vector (v around it ₀and v ₁-v ₈) mean value; D _nrepresent v _mand v ₀and the average distance of eight neighborhood block motion vectors; D _cthen represent and treat inserted block v ₀and v _mdistance.

By judging D _nand D _cnumerical values recited, thus judge block motion vector v to be inserted ₀reliability: if D _c> D _n, then v is thought ₀be unreliable motion vector, namely need v ₀smoothing.

Step3. judge the reliability of the motion vector of eight the neighborhood blocks treated around inserted block, reliable neighborhood block motion vector, as described in Step2, is put into set T by determination methods;

Step4. utilize the reliable neighborhood block motion vector obtained in Step3 to unreliable motion vector v ₀smoothly, adopt based on the weighting based on distance level and smooth, its weighted value is determined by the distance treated between inserted block and neighborhood block (Fig. 6 (e)), as shown in Equation 9:

$w_i=\left\{\begin{array}{cc}1,& i=2,4,5,7\\ \frac{1}{\sqrt{2}},& i=1,3,6,8\end{array}\right.$

(formula 9)

$v_{0m}=\frac{\underset{i\∈T}{\Σ}{w}_i\×v_i}{\underset{i\∈T}{\Σ}{w}_i}$

Note: v in above formula _irepresent i-th motion vector, then w in the 3*3 vector extrapolation figure after sequence _irepresent v _iweighted value, T represents the set of block vector around the reliability that obtains in Step3, v _0mrepresent the motion vector treating inserted block after weighting smoothly.

(6) bi directional motion compensation: in order to effectively solve the blocking effect problem that block-based motion estimation brings, overlapping block bi directional motion compensation is adopted to carry out interleave, as shown in Figure 7, wherein, solid line representative be the original coupling of N × N, B representative is current block, and N1-N8 represents block around current block; Dashed boxes centered by nine blocks above respectively to the extension blocks of upper and lower, left and right four direction expansion w width formation (N+2w) × (N+2w).

Employing standard yuv video cycle tests Foreman sequence emulates, and has done subjectivity and objectivity compare with the simulation result of traditional frame per second method for improving based on three limit filtering, the frame per second method for improving based on medium filtering.As shown in Figure 8, for the comparison diagram of Foreman sequence, figure (a)-(d) represents the methods and results figure of former figure, three limit filtering method result figure, median filter method result figure, the application respectively, and figure (e) is then to 23 frame emulation gained Y-PSNR PSNR (PeakSignaltoNoiseRatio) comparative graph before Foreman sequence.No matter be subjective results or objective evaluation, can find out that the method for this patent is all significantly improved relative to existing method, efficiently solve the cavity of interpolation frame, overlap, block and the problems such as motion blur.

Claims (4)

1. based on a frame per second method for improving for motion vector reprocessing, it is characterized in that: at motion estimation stage, no longer carry out unidirectional estimation, but have employed the motion estimation scheme that three different directions combine; In motion vector post-processing stages, no longer adopt unified medium filtering, but adopt different self adaptation vector corrections and Smooth scheme for the motion vector of different directions, diverse location, concrete steps are:

Step 1: process original video, is treated to frame;

Step 2: the forward and backward estimation based on diamond block search is carried out respectively to two continuous frames;

Step 3: to estimating that the forward and backward motion vector obtained carries out the adaptive motion vector correction based on prediction respectively;

Step 4: select the vector of optimum orientation as two-way candidate motion vector in revised forward and backward motion vector;

Step 5: location-based adaptive motion vector carries out to two-way candidate motion vector level and smooth;

Step 6: two-way overlapped block motion compensation obtains inserting frame;

Step 7: video insertion frame and primitive frame being synthesized high frame per second.

2. the frame per second method for improving based on motion vector reprocessing according to claim 1, it is characterized in that: first rejecting outliers is carried out to forward and backward motion vector in step 3, then the adaptive motion vector correction based on prediction is adopted to exceptional value block vector, to reject vector exceptional value, improve the accuracy of one-way movement vector.

3. the frame per second method for improving based on motion vector reprocessing according to claim 1, it is characterized in that: in step 4 respectively based on absolute error and the sad value of forward and backward motion vector computation match block and current block, choosing the little motion vector of corresponding sad value is two-way candidate motion vector, thus revises the inaccurate motion vector because occlusion issue causes.

4. the frame per second method for improving based on motion vector reprocessing according to claim 1, it is characterized in that: in steps of 5 according to treating inserted block diverse location, adaptive motion vector carries out to two-way candidate motion vector level and smooth, concrete grammar is: edge blocks and the advanced row vector extrapolation of four hornblock vectors, then adopts the weight vectors based on distance level and smooth; Horizontal direction and the judgement of vertical direction scope are first carried out to intermediate mass, to determine the candidate vector treating inserted block, the sad value treating current block in match block and present frame in the search frame that inserted block is corresponding is calculated respectively again based on candidate vector, choosing the minimum motion vector of corresponding sad value is optimal motion vectors, revises inaccurate motion vector.