CN103888767B - A kind of frame per second method for improving that UMH block-based motion estimations are combined with optical flow field estimation - Google Patents

Abstract

The invention provides a method for increasing the frame rate. The method is mainly divided into four steps, which are image segmentation to obtain foreground, background and object edges; variable-size block matching motion estimation is used for the foreground and background, and optical flow field-based motion estimation is used for object edges; motion vector Perform post-processing to obtain reliable motion vectors; use overlapping block motion compensation and bilinear interpolation methods for motion compensation, and synthesize and insert frames. The method for increasing the frame rate proposed by the present invention can solve the problems such as the halo effect and the edge block jagged effect in the traditional method for increasing the frame rate, and is widely used in the field of increasing the frame rate.

Description

A frame rate improvement based on the combination of UMH block matching motion estimation and optical flow field motion estimation liter method

technical field

The invention relates to a method for increasing frame rate video, which belongs to the field of video data processing.

Background technique

The frame rate improvement method of video is to improve the visual quality of low frame rate video and obtain high frame rate video by inserting predicted frames into the original frame rate video sequence. Due to the diverse applications of video frame rate boosting, frame rate boosting technology is becoming more and more important in the field of consumer electronics. HDTV and multimedia PC systems can play video with a higher frame rate than broadcast video streams, and video frame rate enhancement technology can be applied to increase the original video frame rate to improve the viewing effect of end users.

At present, the most widely used method for video frame rate improvement is the motion compensation method based on motion estimation, and most of the frame rate improvement methods are motion estimation methods based on block matching. However, at the junction of the foreground and the background, due to the inaccuracy of the edge block matching of the moving object, the inaccurate motion vector is estimated, so that problems such as halo effect and jagged edge will appear in the occluded area, resulting in the increase of the frame rate. The video quality of the video is degraded, affecting the visual effect of the end user.

Contents of the invention

Aiming at the problems of halo effect and jagged edges in the frame rate improvement, this application provides a method for moving object detection to separate the foreground and background, and adopts a video frame rate improvement method based on the combination of block matching based motion estimation and optical flow field motion estimation , to improve video quality after frame rate boosting.

Technical solution of the present invention is as follows:

A video frame rate promotion method based on UMHexagonS block matching motion estimation and optical flow field motion estimation is characterized in that the method comprises the following steps:

Step 1: Process the original video, use the inter-frame difference method to separate the foreground and background, and mark the edge pixels;

Step 2: use variable size UMHexagonS block matching motion estimation to obtain the motion vectors of foreground and background;

Step 3: Using optical flow field motion estimation to obtain the motion vector of the pixel point on the edge of the moving object;

Step 4: post-processing the obtained motion vector;

Step 5: Carry out overlapping block motion compensation on the foreground and background, and perform bilinear interpolation motion compensation on the edge of the object to obtain the interpolation frame;

Step 6: Combine the inserted frame and the original frame into a high frame rate video.

Preferably, in step 2 and step 3, an adaptive motion estimation method is used for the foreground, background, and object edge respectively, so as to improve the accuracy of the motion vector of the pixel point of the object edge.

Preferably, in step 4, a reliability judgment is performed on the obtained motion vectors, and a median filter is performed on unreliable motion vectors, so as to improve the accuracy of the motion vectors.

Preferably, motion compensation for overlapping blocks is performed in step 5 to reduce block effects and improve video quality.

Description of drawings

Fig. 1: is the overall processing block diagram of the present invention.

Figure 2: Schematic diagram of UMHexagonS block matching motion estimation method.

Figure 3: Simulation result graph.

detailed description

The present invention is aimed at the movement of objects in the image, which is divided into foreground, background and edge regions, respectively adopts a size-variable block matching motion estimation method, and a method based on optical flow field motion estimation, and adopts overlapping block motion compensation after post-processing the motion vector The method constructs the inserted frame to reduce the effect of halo effect and edge aliasing, and achieve the goal of reconstructing high-quality and high-frame-rate video.

The present invention will be further described below in conjunction with specific embodiments (but not limited to this example) and accompanying drawings.

(1) Processing of original digital images:

(1) read in the video;

(2) Counter t=1 is set, and the i-th frame is saved successively as the current frame, the t+2 frame is used as the next frame, and the t+1 frame is reserved as the frame to be inserted;

(3) Using dynamic adaptive threshold inter-frame difference method to detect moving objects to separate foreground and background. Motion detection based on frame difference is the frame difference method, which detects moving objects according to the brightness change between adjacent frames or every other frame image. Specifically include the following steps:

a. Calculate the difference between the t-th frame image and the t+2-th frame image by formula (1), and record it as D(x,y):

D(x,y)＝|F _t+2 (x,y)-F _t (x,y)| (Formula 1)

Among them: F _t (x, y), F _t+2 (x, y) represent the images at time t and t+2 respectively;

b. The selection of the dynamic adaptive threshold TH, the value of the threshold TH is 1/2 of the difference between the highest gray value and the lowest gray value of the difference image in the frame difference method. Through step a, record the maximum value of D(x,y) as D _max , the minimum value as D _min , TH=(D _max -D _min )/2;

c. Binarize the difference image D(x, y), and perform image segmentation according to formula 2:

In formula 2, R(x, y) is the difference image after binarization, and TH is the threshold of image segmentation.

(2) Estimation stage of motion vector:

(1) As shown in Figure 2, for moving objects, the UMHexagonS bidirectional motion estimation method with a block size of 4*4 is used. Specific steps are as follows:

a. The t+1th frame of the virtual frame to be inserted, according to the segmentation result of step (1), the position of the foreground object is divided into several 4*4 rectangular blocks, and two-way motion estimation is carried out to each small block in turn;

b. The criteria for block matching are as follows: Calculate the minimum absolute error sum (Sum of Absolute Difference, SAD) of the blocks corresponding to the tth frame and the t+2th frame according to formula 3, and SAD is defined as follows:

Among them, B _{i, j} is the block to be estimated in the t+1th frame, v is the candidate motion vector, s is the pixel to be inserted in the t+1th frame, f _t [sv] is the t+1th forward Mapped to the corresponding pixel in the tth frame, f _t+1 [s+v] is the backward mapping of the t+1th frame to the corresponding pixel in the t+2th frame;

c. According to the matching criteria, the Unsymmetrical Multi-resolution Hexagon (UMHexagonS) search method uses a mixed multi-level search method, and the specific steps are as follows:

Step1: For the prediction of the center point, the initial center point is firstly predicted by the median value, then the upper layer prediction, and finally the motion vector prediction of the corresponding block of the previous frame;

Step2: Hybrid multi-level motion search;

Step2.1 Asymmetrical cross search, then conduct a grid search with the searched optimal point as the center point, and then conduct a large hexagonal search with the current optimal point as the center point;

Step2.2 Carry out extended hexagon search until the best point is the center point or stop when the maximum number of searches is reached;

Step2.3 Narrow down the search range and perform a diamond search until the optimal point is the center point or the maximum number of searches is reached.

(2) For a relatively still background, adopting a UMHexagonS with a block size of 8*8 to search for a two-way motion estimation method (the specific steps are as in step (1));

(3) For the edge contours of the foreground and background, the motion estimation based on the optical flow field method is adopted. The basic model of optical flow calculation is: assuming that the brightness value of the pixel on a small spatial neighborhood remains constant, the local optical flow field of the moving image can be calculated quickly by the least squares optimization method. The specific steps are as follows:

a. Let f(x, y, t) represent the continuous spatio-temporal brightness distribution, if the brightness remains constant along the trajectory we can get:

In Formula 4, x and y change along the trajectory with time t respectively. Applying the differential chain rule to Equation 4 gives:

In formula 5, represent the components of the motion vector along the space coordinates, respectively. Equation 5 is called optical flow equation or optical flow constraint, and Equation 5 can also be written in the form of <vector inner product>:

b. The minimum value of the pixel-by-pixel variation of the flow vector of the sports field should satisfy the optical flow equation, so that:

Equation 7 represents the error in the optical flow equation, and when ε _of (v(x,y,t)) is equal to 0, the optical flow equation is satisfied. In the presence of occlusion and noise, find the minimum value of the square of ε _of (v(x,y,t)). We can use the regularization method to find the optical flow, so that the following formula is minimized:

(Formula 8)

In formula 8, is the λ Lagrangian multiplier, if the derivative ▽f and If it can be obtained more accurately, the larger parameter can be selected, otherwise the smaller one can be selected.

(3) Motion vector post-processing stage:

a. Judgment of motion vector reliability:

Step1: Calculate the average value of the motion vectors of the block to be judged (denoted as block B) and its surrounding eight blocks:

In the formula, v _m is the average value, and v _i represent the motion vectors of the surrounding eight blocks respectively.

Step2: Calculate the average difference:

In the formula, v _m is the average value, and v _i represents the motion vector of the B block.

Step3: Calculate the difference:

Dc=|v _m -v ₁ | (Formula 11)

Step4: Judgment, if Dc>Dn, then v ₁ is an unreliable motion vector and requires median filtering.

b. Median filtering of unreliable motion vectors:

v _1smooth = median[v ₁ ,v ₂ ,v ₃ ,...,v ₉ ] (Formula 12)

(4) Motion compensation stage: use overlapping block motion compensation method (OBMC) for foreground and background objects, and adopt bilinear interpolation motion compensation method for foreground and background edges. When the motion vector estimation is inaccurate or the motion of the object is not a simple translation motion and there are many different objects in a block, the overlapping block motion compensation method can solve the block effect problem. With the OBMC method, the prediction of a pixel is not only based on the estimation of the motion vector of the block to which it belongs, but also based on the estimation of the motion vector of the neighboring blocks.

In the traditional video frame rate improvement method, when performing block-based motion estimation on the edge of the moving object, the pixels of the background are used, which causes the inaccuracy of the edge estimation. Using pixel-based motion estimation on the edge of the moving object will not have the problem of using background pixel information to estimate the pixel information of the moving object, so that the correct motion vector can be obtained, which effectively solves the problem of edge halo effect and jagged blocks.

As shown in Figure 3, from left to right and from top to bottom in the figure are UMHexagonSexagonS block matching motion estimation direct motion compensation, overlapping block motion compensation, optical flow field motion estimation bilinear interpolation motion compensation, this patent motion Compensation simulation results.

The present invention adopts the standard YUV video test sequence foreman sequence to obtain the simulation results, and compares it with the UMHexagonSexagonS motion estimation compensation method, the optical flow field motion estimation interpolation method, and the overlapping block motion compensation method. It can be seen that the method of the present invention effectively solves the problem of Edge halo effect, edge jagged block issues.

Claims (3)

1. A video frame rate promotion method based on UMHexagonS block matching motion estimation and optical flow field motion estimation is characterized in that the method comprises the following steps: Step 1: Process the original video, use the inter-frame difference method to separate the foreground and background, and mark the edge pixels; Step 2: use variable size UMHexagonS block matching motion estimation to obtain the motion vectors of foreground and background; Step 3: Using optical flow field motion estimation to obtain the motion vector of the pixel point on the edge of the moving object; Step 4: Post-processing the obtained motion vector, the specific steps are: a. Judgment of motion vector reliability: (1): Calculate the average value of the motion vectors of the block to be judged, that is, block B and its surrounding eight blocks: ${\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 9</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 9</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$ In the formula, v _m is the average value, and v _i represent the motion vectors of block B and its surrounding eight blocks respectively, wherein v ₁ is the motion vector of block B; (2): Calculate the average difference: ${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 8</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 9</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; ;</span>$ (3): Calculate the difference: D _C =|v _m -v ₁ |; (4): Judgment, if D _C >D _n , then v ₁ is an unreliable motion vector, requiring median filtering; b. Median filtering of unreliable motion vectors: v _1smooth = median[v ₁ ,v ₂ ,v ₃ ,...,v ₉ ]; Step 5: Carry out overlapping block motion compensation on the foreground and background, and perform bilinear interpolation motion compensation on the edge of the object to obtain the interpolation frame; Step 6: Combine the inserted frame and the original frame into a high frame rate video. 2. The video frame rate promotion method based on UMHexagonS block matching motion estimation and optical flow field motion estimation according to claim 1, is characterized in that in step 2 and step 3, adaptive A motion estimation method to improve the accuracy of motion vectors of pixel points on the edge of an object. 3. The video frame rate improvement method based on UMHexagonS block matching motion estimation and optical flow field motion estimation according to claim 1, characterized in that in step 5, overlapping block motion compensation is performed to reduce block effects and improve video quality .