CN104065975A - Frame rate up-conversion method based on adaptive motion estimation - Google Patents

Abstract

The invention provides a frame rate up-conversion method which comprises the following four steps: adaptive motion estimation is performed by adoption of a texture classification method based on edge detection to obtain motion vectors; motion vector post-processing is performed, namely, inaccurate motion vectors are corrected by adoption of a multi-direction extent motion vector repairing method; motion vectors of a hole area and an overlapped block area are estimated; and motion compensation is performed according to the obtained motion vectors to synthesize an insert frame. By adopting the frame rate up-conversion method provided by the invention, the problems such as motion blur, block effect, hole and overlapped block caused by the traditional frame rate up-conversion method can be effectively solved, and the quality of synthetic videos is greatly improved.

Description

The frame per second method for improving of estimating based on adaptive motion

Technical field

The present invention relates to a kind of method that frame-rate video promotes, belong to video data process field.

Background technology

It is a kind of video post-process method that promotes video quality by improving video frame rate that frame per second promotes.Due to the restriction of liquid crystal display self hardware condition, when playing the video that moves violent, there will be motion blur and motion jitter phenomenon, be called as ghost effect, can have a strong impact on terminal use's viewing effect.Frame per second lift technique can effectively reduce ghost effect by promoting original video frame per second, improves video quality; Under the condition being restricted in channel width, at coding side, must reduce transmitted data amount, only transmit a part of video content, in decoding end, by frame per second lift technique, can recover complete video content so, both promoted the utilance of channel width, guaranteed again that video quality can meet user's viewing demand.Because video frame rate promotes diversified application, frame per second lift technique is more and more important in consumer electronics field.HDTV and multimedia PC system can be play the video higher than broadcast video stream frame per second, and video frame rate lift technique just can be applied to lifting original video frame per second and improve terminal use's viewing effect.

The current method that mainly contains two class frame per second liftings, a kind of method is not considered the movable information between frame, is only to obtain inserting frame by related pixel value linear combination between consecutive frame, typical method comprises: frame repetition and frame are average; Another kind of method is based drive method, and most frame per second method for improving is block-based motion estimation and compensation all.Based drive frame per second method for improving can effectively improve the quality of video, but has also produced problems such as motion blur, blocking effect, cavity, overlapping block simultaneously.

Summary of the invention

For solving the empty problem occurring in frame per second lifting, the application provides a kind of method that motion is repaired based on hole region, according to the motion vector estimation of inserting cavity piece around the correspondence position of reference frame in frame, go out the motion vector of hole region, carry out motion compensation filling, the method of only filling with reference frame pixel value, adds the empty repairing effect of movable information better; For obtaining motion vector accurately, to solve motion blur, the blocking effect problem occurring in frame per second lifting, the application provides a kind of adaptive movement estimation method, according to the textural characteristics of reference frame, divide different macro blocks, the piece not abundant to texture adopts overlapping block motion estimation algorithm, by expanding the size of piece, carries out the probability that estimation reduces inaccurate motion vector appearance; The piece of texture-rich is adopted to first motion algorithm for estimating; For further obtaining correct motion vector, in motion vector post-processing stages, introduce motion vector angle absolute difference and absolute error and (SAD, Sum of Absolute Differences) motion vector is classified, inaccurate motion vector adopts multi-direction extension blocks motion vector restorative procedure, obtains optimal motion vectors.

In the present invention, first present frame is carried out to rim detection, according to each piece, comprise that the number of edge pixel point divides the type of macro block, be divided into the not abundant piece of the piece of texture-rich and texture; Then, the piece not abundant to texture carries out overlapping block estimation, and the piece of texture-rich is carried out to first motion estimation; Afterwards, utilize motion vector angle absolute difference and sad value to divide motion vector kind, inaccurate motion vector moves to repair and obtains motion vector accurately; Finally, hole region and overlapping region are carried out to estimation, utilize to insert hole region same position piece motion vector estimation around in former frame in frame and go out the motion vector of hole region, and it is carried out to first motion compensation obtain inserting frame.

Technical solution of the present invention is as follows:

A frame per second method for improving of estimating based on adaptive motion, is characterized in that the method comprises the following steps:

Step 1: original video is processed, be treated to frame;

Step 2: according to edge detection results, frame is divided into texture-rich piece and texture does not enrich piece, dissimilar piece is carried out to adaptive motion estimation, obtain motion vector;

Step 3: adopt multi-direction extension movement vector restorative procedure, revise inaccurate motion vector;

Step 4: the motion vector of estimating hole region;

Step 5: motion compensation obtains inserting frame;

Step 6: will insert the video of the synthetic high frame per second of frame and primitive frame.

Preferably, in step 2, adopt Sobel Operator (sobel operator) to carry out rim detection and obtain marginal information, add up the number the calculating mean value that in each macro block, comprise edge pixel point, when comprising edge pixel point number being greater than mean value in each macro block, think the piece of texture-rich, otherwise be that texture does not enrich piece; For the not abundant piece of texture, adopt overlapping block estimation, for the piece employing first motion of texture-rich, estimate.

Preferably, in step 3, according to the motion vector angle absolute difference of each piece and absolute error and (SAD, Sum ofAbsolute Differences) carry out motion vector classification, for inaccurate motion vector, respectively the motion vector value increase and decrease 1 of level and vertical direction is formed to nine candidate motion vectors, calculate respectively match block that nine motion vectors are corresponding and the sad value of current block, the motion vector of choosing corresponding sad value minimum is optimal motion vectors, revises inaccurate motion vector.

Preferably, in step 4, for hole region, according to insert empty piece position in frame in former frame correspondence position around the motion vector estimation of piece go out the motion vector of empty piece; For overlapping region, adopt motion vector corresponding to smallest match criterion as optimal motion vectors.

Accompanying drawing explanation

Fig. 1 is disposed of in its entirety block diagram of the present invention.

Fig. 2 is rim detection Sobel convolution factor schematic diagram, and wherein (a) is the horizontal detection factor, is (b) the vertical detection factor.

Fig. 3 is the result exploded view of rim detection, and wherein (a) is the first frame of YUV normal video foreman sequence, (b) is the edge detection results of (a).

Fig. 4 is overlapping block method for estimating schematic diagram, and wherein (a) represents an overlapping block on present frame, (b) represents overlapping block estimation.

Fig. 5 is multi-direction extension blocks motion vector restorative procedure schematic diagram, and wherein (a) represents the extension blocks on present frame, (b) represents the candidate motion vector of nine different directions.

Fig. 6 is that schematic diagram is repaired in hole region motion.

Fig. 7 is Bus video simulation result comparison diagram.

Fig. 8 is Foreman video simulation result comparison diagram.

Embodiment

The present invention enriches degree according to image texture and adopts adaptive motion method of estimation, again by multi-direction extension blocks motion vector restorative procedure, obtain optimal motion vectors, then estimation is carried out in cavity and overlapping block region, by motion compensation, complete insertion frame, effectively solve cavity, overlapping block problem, played the effect that reduces ghost effect, blocking effect, reached the target of the high frame-rate video of reconstruct high-quality.

Below in conjunction with specific embodiment (but being not limited to this example) and accompanying drawing, the present invention is further detailed.

(1) processing to original digital image:

(1) read in video;

(2) counter t=1 is set, preserves successively t frame as present frame, be denoted as f _t; T+2 frame, as next frame, is denoted as f _t+2; Reserved t+1 frame, as incoming frame to be inserted, is denoted as f _t+1;

(2) estimation stages of motion vector:

(1) rim detection based on Sobel Operator (sobel operator):

Sobel operator is a discreteness difference operator, is used for the approximation of gradient of computed image luminance function.In any point of image, use this operator, will produce corresponding gradient vector or its method vector, thereby carry out rim detection.As shown in Figure 2, for the Sobel convolution factor, the matrix of two groups of 3x3 of this operator inclusion, be respectively horizontal and longitudinal, as accompanying drawing 2 (a) is depicted as the horizontal detection factor, 2 (b) vertically detect the factor, and it and image are made to planar convolution, can draw respectively laterally and the approximation of brightness difference longitudinally.If with f _trepresent present frame, G _xand G _yrepresent that respectively its formula is as follows through gradation of image value horizontal and that longitudinal edge detects:

$G_x=\left[\begin{array}{ccc}-1& 0& +1\\ -2& 0& +2\\ -1& 0& +1\end{array}\right]*f_t$ $G_y=\left[\begin{array}{ccc}+1& +2& +1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right]*f_t$ (formula 1)

F _tin the value of each pixel press formula 2 and calculate:

| G|=|G _x|+| G _y| (formula 2)

If gradient G is greater than a certain threshold values, think that this point (x, y) is marginal point, the result that accompanying drawing 3 is rim detection, wherein 3 (a) are the former figure of the first frame of standard yuv video foreman sequence; 3 (b) are the result figure that the first frame border detects.

(2) textural characteristics classification:

The edge of image refers to two edges with the even image-region of different gray scales, and around gray-scale intensity has the set of those pixels that contrast changes; And image texture is the variation of gray scale and color in image in general sense, a connected pixel set that repeats to meet given gamma characteristic in an image-region has formed a texture region, so on definite meaning, can use the texture of edge token image.Edge detection graph according to obtaining in step (1), is denoted as A (x, y); A (x, y) is divided into several pieces of 8 * 8, adds up the number that in each piece, edge pixel point occurs, be denoted as N _{m * n}(x), m wherein, n represents the subscript of each piece; Calculate all mean values that comprise edge pixel point number, be denoted as ave (x); If N _{m * n}(x) > ave (x), thinks that this piece is the piece of texture-rich, otherwise thinks the piece that texture is not abundant.

(3) adaptive motion estimation:

In the application, adopt asymmetric cross multi-level hexagonal point search (Unsymmetrieal-CrossMuti-HexagonSearch) block matching motion estimation method, the method belongs to a kind of rapid motion estimating method, operand, only less than 10% of full-search algorithm, can keep again higher estimation accuracy simultaneously; , as matching criterior, computational methods are shown in formula 3, formula 4 in the application, to adopt absolute error and (SAD, Sum ofAbsolute Differences);

$\mathrm{SAD}(\mathrm{dx},\mathrm{dy})=\underset{x\∈B_x}{\mathrm{\Σ}}\underset{y\∈B_y}{\mathrm{\Σ}}|f_t(x,y)-f_{t+2}(x+\mathrm{dx},y+\mathrm{dy})|$ (formula 3)

$v=\arg \underset{\mathrm{dx},\mathrm{dy}}{\mathrm{MIN}}\mathrm{SAD}(f_t,f_{t+2};\mathrm{dx},\mathrm{dy})$ (formula 4)

Wherein, (x, y) is f _tpixel to be matched in frame; (dx, dy) is f _t+2candidate's pixel in frame; B _x, B _yrepresent respectively the length and width scope of macro block, the optimal motion vectors that v representative estimates, optimal motion vectors is the motion vector that minimum sad value is corresponding.

According to the Texture classification in step (2), for dissimilar piece, adopt adaptive movement estimation method.For f _tthe not abundant piece of texture in frame, easily at f _t+2frame finds more match block, but these match block are best matching blocks not necessarily, so adopt overlapping block method for estimating, expand to 16 * 16 piece and calculate optimal motion vectors as match block, in these overlapping blocks, may comprise some important texture recognition information, improve the accuracy of estimation.As shown in Figure 4, region representation f shown in left slash in 4 (a) _tthe not abundant piece of texture in frame, region shown in right slash is the overlapping block of expansion, 4 (b) represent the process of unidirectional overlapping block estimation; For the piece of texture-rich, mostly belong to edge block, adopt the little match block of 8*8 can refinement edge object, but reduce due to two be different objects sad value minimum error probability again;

(3) motion vector post-processing stages:

(1) bilateral filtering of motion vector:

A. the judgement of motion vector reliability:

Step1: calculate and to want decision block (being designated as B piece) and the mean value of the motion vector of eight pieces around thereof:

$v_m=\frac{1}{9}\underset{i=1}{\overset{9}{\mathrm{\Σ}}}{v}_i$ (formula 5)

V in formula _mfor mean value, v _irepresent respectively the motion vector of eight pieces around.

Step2: calculated difference:

$D_n=\frac{1}{8}\underset{i=2}{\overset{9}{\mathrm{\Σ}}}|v_m-v_i|$ (formula 6)

V in formula _mfor mean value, v ₁represent the motion vector of B piece.

Step3: calculate mean difference:

D _c=| v _m-v ₁| (formula 7)

Step4: judgement, if D _c> D _n, v ₁for unreliable motion vector, need bilateral filtering.

B. unreliable motion vector is carried out to bilateral filtering:

V _1smooth=bfilter2[v ₁, v ₂, v ₃..., v ₉] (formula 8)

(2) classification of the motion vector based on motion vector differential seat angle and sad value:

If the angle difference of the angle of a piece motion vector and around piece motion vector is little, think that so the motion vector of this piece is relatively accurate; If the matching criterior sad value of the motion vector of a piece is relatively little, think that so the motion vector of this piece is also relatively accurate, therefore can utilize the absolute difference of motion vector angle and the accuracy that sad value judges this block motion estimation; Adopt the absolute difference of formula 9 calculation of motion vectors angles, adopt formula 10 to calculate normalized absolute error and SAD (Sum of Absolute Differences):

${\mathrm{Angle}}_{\mathrm{mean}}(F,N)=|\mathrm{Angle}\left(T\right)-\frac{1}{4}\underset{n=1}{\overset{4}{\mathrm{\Σ}}}\mathrm{Angle}\left(N_n\right)|$ (formula 9)

Wherein, Angle _mean(F, N) represents f _tcurrent block and the absolute difference of the motion vector angle of four pieces around in frame, Angle (T) represents f _tthe motion vector angle of current block in frame, Angle (N _n) represent f _tthe motion vector angle of four pieces in current block upper and lower, left and right in frame.

${\mathrm{SAD}}_{\mathrm{mean}}(F,R)=\frac{{\mathrm{\Σ}}_{i,j}|T(i,j)-R(i,j)|}{{\mathrm{\Σ}}_{i,j}T(i,j)}$ (formula 10)

SAD wherein _mean(F, N) represents f _tthe normalization SAD of current block in frame, T (i, j) represents f _tall pixel values of current block in frame, R (i, j) represents f _t+2all pixel values of match block in frame.By calculating normalized SAD, can in the scope of 0-1, divide motion vector, it is more accurate to make to divide.

The criteria for classification of motion vector:

If Angel a. _mean(F, N) < A ₁, judge that this motion vector is accurately;

If A b. ₁≤ Angle _mean(F, N)≤A ₂, then judge SAD _mean(F, R);

If SAD c. _mean(F, R) > B, judges that this motion vector is inaccurate.

A wherein ₁, A ₂, B is threshold value, is all according to the histogram of the absolute difference of motion vector angle and normalized SAD, definite threshold value.In this application, A ₁=0.45, A ₂=1.8, B=0.06.

(3) reparation of inaccurate motion vector:

The motion vector classification results drawing according to step (2), inaccurate motion vector adopts multi-direction extension blocks motion vector restorative procedure, corrects inaccurate motion vector.For in corresponding of inaccurate motion vector, may comprise a plurality of moving object, just can make motion vector absolute difference and sad value all larger, just can refinement moving object so be divided into fritter by inaccurate, thus improve the accuracy of estimating.Meanwhile, in order to ensure moving object edge integrity, on the basis of fritter, increase extension blocks, also avoided the mistake that local optimum is brought to estimate.As shown in accompanying drawing 5 (a), first, first 8 * 8 of current blocks are divided into four 4 * 4, the fritter that each is divided into expands to 6 * 6 piece; Then the level of motion vector and vertical vector are increased and decreased respectively to 1, form nine groups of candidates' motion vector, as shown in Fig. 5 (b), the vector that wherein (a) representative will be revised, (b)---(i) represent respectively upper left, upper, upper right, left, lower-left, under, bottom right, right eight groups of candidate motion vectors, to these nine groups of candidate motion vectors, adopt respectively 6 * 6 to calculate sad values, select motion vector that minimum sad value is corresponding as the optimal motion vectors of revising.

(4) the motion reparation of hole region:

Utilize to insert empty piece position in frame in former frame correspondence position around the motion vector estimation of piece go out the motion vector of empty piece; For overlapping region, adopt motion vector corresponding to smallest match criterion as optimal motion vectors.

(1) hole region estimation: as shown in Figure 6,6 (a) represents f _tframe, 6 (b) represents f _t+1frame, 6 (c) represents f _t+2frame, the piece that left brace is filled represents normal motion estimation block, at f _t+1in the piece that in frame, stain is filled, have cavity, this piece is at f _tin frame, corresponding piece is the piece that stain is filled, and so just gets the intermediate value of motion vector of the piece that stain piece eight right slash around fill as f _t+1the motion vector of empty piece in frame.

(2) hole region motion compensation: as shown in Equation 11, utilize the method for similar bi directional motion compensation to complete the cavity filling of inserting frame:

$f_{t+1}(x,y)=\frac{1}{2}\&times;[f_t(x-\mathrm{dx},y-\mathrm{dy})+f_{t+2}(x+\mathrm{dx},y+\mathrm{dy})]$ (formula 11)

F wherein _t+1for treating the piece that is inserted in interleave, f _tand f _t+2be respectively the match block in former frame and a rear frame.

(5) motion compensation stage: as shown in Equation 12, carry out first motion compensation.

$f_{t+1}(x,y)=\frac{1}{2}\&times;[f_t(x,y)+f_{t+2}(x+\mathrm{dx},y+\mathrm{dy})]$ (formula 12)

This patent adopts standard yuv video cycle tests Bus sequence and Foreman sequence to obtain simulation result, compares with traditional overlapped block motion compensation frame per second method for improving, the frame per second method for improving based on three limit filtering, the frame per second method for improving based on full search.As shown in Figure 7, subjective comparison diagram for Bus sequence, figure (a)-(d) represent respectively former figure, all direction search method result figure, three limit filtering method result figure, the application's methods and results figure, figure (e)-(g) is respectively the details enlarged drawing of three kinds of methods; As shown in Figure 8, subjective comparison diagram for Foreman sequence, figure (a)-(d) represent respectively former figure, all direction search method result figure, three limit filtering method result figure, the application's methods and results figure, figure (e)-(g) is respectively the details enlarged drawing of three kinds of methods; As shown in table 1, it is the objective evaluation comparison of three kinds of methods, the standard of evaluating be Y-PSNR PSNR (PeakSignal to Noise Ratio) and structural similarity SSIM (structural similarity index measurement) can find out this patent method at subjective effect or objective effect has all had obvious lifting, efficiently solve the problems such as cavity, overlapping block, blocking effect.

Table 1: objective simulation result comparison diagram

Claims (4)

1. a frame per second method for improving of estimating based on adaptive motion, is characterized in that the method comprises the following steps:

Step 1: original video is processed, be treated to frame;

Step 2: according to edge detection results, frame is divided into texture-rich piece and texture does not enrich piece, dissimilar piece is carried out to adaptive motion estimation, obtain motion vector;

Step 3: adopt multi-direction extension movement vector restorative procedure, revise inaccurate motion vector;

Step 4: the motion vector of estimating hole region;

Step 5: motion compensation obtains inserting frame;

Step 6: will insert the video of the synthetic high frame per second of frame and primitive frame.

2. the frame per second method for improving of estimating based on adaptive motion according to claim 1, it is characterized in that: in step 2, adopt Sobel Operator (sobel operator) to carry out rim detection and obtain marginal information, add up the number the calculating mean value that in each macro block, comprise edge pixel point, when comprising edge pixel point number being greater than mean value in each macro block, think the piece of texture-rich, otherwise be that texture does not enrich piece; For the not abundant piece of texture, adopt overlapping block estimation, for the piece employing first motion of texture-rich, estimate.

3. the frame per second method for improving of estimating based on adaptive motion according to claim 1, it is characterized in that: in step 3, according to the motion vector angle absolute difference of each piece and absolute error and sad value, carry out motion vector classification, for inaccurate motion vector, respectively the motion vector value increase and decrease 1 of level and vertical direction is formed to nine candidate motion vectors, calculate respectively match block that nine motion vectors are corresponding and the sad value of current block, the motion vector of choosing corresponding sad value minimum is optimal motion vectors, revises inaccurate motion vector.

4. the frame per second method for improving of estimating based on adaptive motion according to claim 1, it is characterized in that: in step 4, for hole region, according to insert empty piece position in frame in former frame correspondence position around the motion vector estimation of piece go out the motion vector of empty piece; For overlapping region, adopt motion vector corresponding to smallest match criterion as optimal motion vectors.