KR20090041562A - Frame interpolating device and frame rate up-converting apparatus having the same - Google Patents

Abstract

A frame interpolating device and a frame rate up-converting apparatus thereof are provided to minimize blurring and other artifacts to preserve an incisive contour of an object. A bilateral motion estimator(12) compares a block shifted from a previous frame and a corresponding position of a block of the next frame. The bilateral motion estimator obtains a motion trajectory passing through one block at an image bilinear interpolation frame. The bilateral motion estimator produces a motion vector of one block in an image bilinear interpolation frame from the information of the next frames and previous. An AOBMC(Adaptively Overlapped Block Motion Compensation) type pixel interpolation(18) uses an adaptive window coefficient, to compensate block motion vectors on the background and object entity radiuses adaptively, interpolates the pixels on the image bilinear interpolation frame from the block motion vectors, and previous and next image frame information.

Description

Frame Interpolating Device and Frame Rate Up-Converting Apparatus having the same}

The present specification relates to an apparatus for up-converting an image frame rate in order to improve temporal resolution of an image frame column.

Typically, the transmission of large amounts of video data over a limited bandwidth channel reduces the spatiotemporal resolution of the video information to achieve the target rate. The transmission speed in the time domain may be reduced by skipping frames, but in this case, the quality of the reconstructed video may be reduced. In order to recover the loss of temporal resolution, an algorithm for upconverting the image frame rate is used in termination processing devices such as an image decoder and a display device.

Frame rate up-conversion algorithms interpolate interpolated video frames between a series of video frames to increase the frame frequency. To this end, the frame rate up-conversion algorithm uses an image frame interpolation method. In the simplest image frame interpolation method, an interpolation frame is generated by combining pixels of the same spatial position without considering object movement. For example, frame repetition or frame averaging belongs to a simple image frame interpolation method. These frame interpolation methods can provide acceptable visual quality in the absence of motion, but can cause artifacts such as blurring and luxuriant movement of the moving object.

In order to reduce such artifacts, motion compensated frame interpolation methods for reconstructing an interpolation frame based on motion information have been proposed. In order to reconstruct a faithful interpolated frame, motion information must be calculated correctly. In other words, unlike video compression, it is most important to obtain authentic motion trajectories in a motion compensated frame rate up-conversion algorithm.

For motion computation, most motion compensated frame interpolation methods use Block Matching Algorithm. This is because the implementation of the block matching algorithm is not only simple and easy, but also the generation of a briefly displayed motion region.

However, the block matching algorithm used in the frame interpolation method of the related art causes the calculated motion vectors often to produce motion vectors that are not faithful to the motion of the authentic object. Indeed, when a block matching algorithm is used in the frame interpolation method of the related art, a plurality of motion trajectories may be lost through one pixel on the interpolation frame or no motion trajectory through one pixel on the interpolation frame. In other words, in the related art frame interpolation method using a block matching algorithm, an overlapping problem as in the former case and / or a hole problem as in the latter case can be caused. In addition, the frame interpolation method of the related art that performs block-by-block processing is difficult to calculate complex motion trajectories when one block on the interpolation frame includes a plurality of objects having different motions.

Because of the difficulty in calculating the authenticity of the motion, the boundary part of the object is deteriorated severely in the image frame interpolated by the frame interpolation method of the related art. In addition, the frame interpolation method and the frame rate up-conversion algorithm of the related art have difficulty in providing a good image frame sequence.

Accordingly, an object of the present disclosure is to provide an embodiment of a frame interpolation apparatus and method capable of interpolating an image frame in which the outline of an object is sharply preserved.

Another object of the present specification is to provide an embodiment of an apparatus and method for frame rate up-conversion capable of up-converting a frame rate of an image without deteriorating a boundary portion of an object.

In accordance with an aspect of the present invention, a frame interpolation apparatus includes: a motion calculator configured to calculate a motion vector for each block of an interpolated image frame through bidirectional comparison using pixel information of a previous image frame and a next image frame; And based on the reliability of the motion vectors of the blocks to be interpolated and adjacent blocks on the interpolated image frame, adaptively adjusting the weighted substitution window coefficients and optionally applying the adaptively adjusted weighted substitution window coefficients. And an adaptive overlapping motion compensated pixel interpolator for performing pixel operation of the image frame frame and the motion vector of the interpolation block to interpolate pixels on the interpolated image frame.

The frame interpolation apparatus includes: an object discriminator for classifying a boundary area between objects on the interpolated image frame based on the block-by-block motion vectors from the bidirectional motion calculator; And reconstruct the motion vector of the boundary block included in the boundary area distinguished by the object discriminator among the block motion vectors from the bidirectional motion calculator into sub-block motion vectors, so that the sub-block motion vectors and the block motion A variable block motion compensator for supplying variable block motion vectors including the vectors to the adaptive overlapping block motion compensated pixel interpolator may be further provided.

The adaptive overlapping block motion compensated pixel interpolator does not apply the adaptively adjusted weighted substitution window coefficient to pixel interpolation when the reliability of the motion vector of the neighboring block is "0".

In accordance with another aspect of the present invention, a frame interpolation method includes: calculating a motion vector for each block of an interpolated image frame through bidirectional comparison using pixel information of a previous image frame and a next image frame; Calculating a reliability of a motion vector of a block to be interpolated and an adjacent block on the interpolated image frame; Adjusting a weighted substitution window coefficient based on the reliability of the motion vector of the adjacent block; And calculating pixel information of the previous and next image frame frames and the motion vector of the interpolation block by selectively applying the adjusted weighted substitution window coefficient according to the reliability of the neighboring block. Interpolating.

The calculating of the motion vector may include: classifying a boundary area between objects on the interpolated image frame based on the motion vectors for each block; And reconstructing a motion vector of a boundary block included in the boundary region among the block motion vectors into sub-block motion vectors.

In accordance with another aspect of the present invention, a frame rate upconversion device includes: a delayer configured to delay an image frame of a first frame frequency from an input line for a period of a first frequency frame; A motion calculator for calculating a motion vector for each block of the interpolated image frame by bidirectional comparison using pixel information of the image frame from the delayer and the image frame from the input line; The input line and the adaptively adjusted weighted substitution window coefficients are adaptively applied based on the reliability of the motion vector of the adjacent block and the block to be interpolated on the interpolated image frame. An adaptive superimposed motion compensated pixel interpolator for performing an operation on the pixel information of the image frames from the delayer and the motion vector of the interpolation block to interpolate pixels on the interpolated image frame; And delayed image frames of which the delayed image frame from the delayer and the interpolated image plane from the adaptive overlapping block motion compensated pixel interpolator are raised to a second frame frequency higher than the first frame frequency and whose frame frequency is increased. And a rate up and combination unit for inserting-combining the interpolated image frame with the frame frequency up.

The second frame frequency may be set to any one of 1.5 times and 2 times the first frame frequency.

According to an embodiment, there is provided a frame rate up-conversion method comprising: delaying an image frame of a first frame frequency on an input line for a period corresponding to a period of a first frame frequency; Calculating a motion vector for each block of the interpolated image frame through bidirectional comparison using pixel information of the image frame and the delayed image frame on the input line; Calculating a reliability of a motion vector of a block to be interpolated and an adjacent block on the interpolated image frame; Adjusting a weighted substitution window coefficient based on the reliability of the motion vector of the adjacent block; According to the reliability of the neighboring block, the pixel information of the previous and next image frame frame and the motion vector of the interpolation block are applied by selectively applying the adjusted weighted substitution window coefficient to interpolate pixels on the interpolated image frame. Making; Increasing the delayed image frame and the interpolated image plane to a second frame frequency that is higher than the first frame frequency and inserting-combining the interpolated image frame with the frame frequency upwards between the delayed image planes with the frame frequency upwards; Include.

According to the above configuration, the frame interpolation apparatus and method of the embodiment can provide a high quality interpolation image frame in which the outline of an object is sharply preserved while minimizing occlusion artifacts. The apparatus and method for up-converting the frame rate of an embodiment may up-convert an image frame sequence to a frame rate while minimizing deterioration of occlusion artifacts and outlines of objects. In addition, the frame interpolation apparatus and method of the embodiment and the frame rate up-conversion apparatus and method of the embodiment can improve the signal-to-noise ratio of the image frame by approximately 2 dB.

In addition to the above embodiments, other objects, other features and advantages of the present specification will become apparent from the detailed description of the embodiments associated with the accompanying drawings.

Before describing the embodiments, the frame interpolation algorithms of the related art will be described.

The quality of an image frame interpolated by a motion compensated frame interpolation algorithm is mainly related to the method of accurately calculating motion trajectories throughout the interpolation frame and to merging motion compensated blocks without non-natural (or non-essential) defects. Depends. In the case of a motion compensated frame interpolation algorithm using a block matching scheme, the skipped (intermediate or interpolated) frame is divided into blocks, and each block is interpolated using information of the previous frame and the next frame. Assume that "f _{n -1} ", "f _n " and "f _{n +1} " represent previous, middle (or interpolated) and next frames, respectively. Further, suppose there is a two-dimensional vector s representing the pixel position. Then, the interpolation intensity (force) of the s-th pixel at f _n is given by Equation 1 below.

f _n [s, v] = (f _{n -1} [sv] + f _{n +1} [s + v]) / 2

Here, "v" is the calculated motion vector for the pixel to be interpolated, and "f _n [s]" represents the original intensity (gradation value) of the s-th pixel Pix on f _n which is the nth frame. For interpolation of a frame using Equation 1, the motion vector v for each interpolation pixel in the interpolation frame f _n must be calculated. The motion vectors v for the interpolation frame f _n are calculated by a three-step process. First, all motion vectors for the frame f _{n to} be interpolated are reset to "0". This is known as the Frame Hold technique. Subsequently, the motion vectors of the interpolated frame f _n of "0" are replaced with the motion vectors of the previous frame f _{n -1} . Finally, block-by-block motion vectors of the interpolated frame are obtained by linear combination of candidate motion vectors between previous and next frames. Although the block-by-block motion vector region between the previous frame and the next frame is obtained by the block matching method, among the pixels on the interpolation frame, the regions of the hole to which the motion trajectories are not applied or the overlapping region where the multiple motion trajectories pass through the same pixel There is no choice but to occur. In other words, block-by-block motion vectors cannot represent complex movements of objects such as magnification, rotation, and eccentricity (deformation). Furthermore, serious occlusion artifacts may be caused because the boundaries of the object do not coincide with the boundaries of the block or have motion vectors that are significantly different from adjacent blocks.

To overcome these problems, various approaches have been proposed to reduce occlusion artifacts by expressing complex movements. First, there is a frame interpolation algorithm that additionally uses spatial transform. In this frame interpolation algorithm, motion vectors for a Control Grid Point are calculated and then interpolated to obtain smooth motion vector regions. Next, a frame interpolation algorithm has been proposed to additionally use overlapping block motion compensation to overcome the problem of overlapping. The overlapped block motion compensated frame interpolation algorithm predicts the interpolated frame by positioning the overlapped block from the reference frame using a weighted displacement window.

As such, even in the frame interpolation algorithm of the related art using the spatial transform method or the overlapping block motion compensation method together with the block matching method, motion compensation errors still occur. As a result, occlusion artifacts and deterioration of the object boundary appear in the interpolation frame, and the quality of the interpolation frame is inevitably low.

Next, embodiments of a frame interpolation apparatus capable of interpolating high quality image frames while minimizing deterioration of occlusion artifacts and object boundary parts and a frame rate up-conversion apparatus including the same will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram schematically illustrating a frame rate up-conversion device according to an embodiment capable of up-converting an image frame rate while minimizing degradation of occlusion artifacts and object boundary portions. The frame rate up-conversion device of FIG. 1 includes a first retarder 10 connected in series with an input line 11, a bilateral motion estimator 12, an object separator 14, Variable-Size Block Motion Compensator 16, and Adaptive Overlaped Block Motion Compensation-type pixel interpolator 18 are provided. The input line 11 is connected to a video source (for example, any one of a video encoder, a demodulation module of a television receiver, a graphics module of a computer system, etc.) not shown. The input line 11 is input with a video frame column of the first frame frequency. The first retarder 10 delays the column of video frames on the input line 11 for the duration of the first frame. For example, if the column of video frames on input line 11 has a frequency of 60 Hz (i.e., video data that causes 60 images to be displayed in one second), the first retarder 10 is 1/60. Delay the first video frame string for a period of seconds. Accordingly, when the image frame delayed by the first delay unit 10 is referred to as the "previous image frame f _{n -1} ", the image frame on the input line 11 is referred to as "next image frame f _{n +1."} ) ". In addition, an image frame to be added between the previous and next image frames f _{n -1} and f _{n +1} becomes an "interpolated image frame f _n ". For example, when two image frames such as FIGS. 2A and 2B sequentially input the input line 11, the image frame of FIG. 2A is the first delay unit 10 as the previous image frame f _{n -1} . 2b is input to the input line 11 as the next image frame f _{n +1} .

The bidirectional motion calculator 12, the object discriminator 14, the variable block motion compensator 16 and the AOBMC type pixel interpolator 18 use previous and next image frames f _{n -1} , f _{n +1} . To construct an "image frame interpolation unit" for generating an interpolated image frame f _n . The image frame interpolator generates an interpolated image frame f _n using the previous image frame f _{n -1} from the first delayer 10 and the next image frame f _{n +1} on the input line 11. do. For example, when the previous image frame f _{n -1} as shown in FIG. 2A and the next image frame f _{n +1} as shown in FIG. 2B are supplied from the first delayer 10 and the input line 11, The image frame interpolator generates an interpolated image frame f _n as illustrated in FIG. 2F.

The bidirectional motion calculator 12 included in the image frame interpolation unit uses the information of the previous and next image frames f _{n -1} and f _{n +1} to obtain motion vectors for each block of the interpolated image frame f _n . Calculate. However, since the interpolated image frame f _n does not have the original pixel values, it is not easy for the bidirectional motion calculator 12 to calculate the motion vectors for the blocks on the interpolated image frame f _n . In order to calculate the block motion vectors, the bidirectional motion calculator 12 uses a bidirectional comparison method for motion calculation to obtain motion vectors using information of previous and next image frames f _{n -1} and f _{n +1} . Adopt.

In detail, the bidirectional motion calculator 12, as shown in FIG. 3, corresponds to a block of the shifted position in the previous image frame f _{n -1} and a corresponding position in the next image frame f _{n +1} . Are compared to obtain a motion trajectory via one block in the interpolated video frame f _n . In FIG. 3, "s" represents one pixel on the interpolated video frame f _n , and "v" represents a candidate motion vector of the pixel. Then, the pixel s on the interpolated image frame f _n is mapped to the rear of the pixel s _b on the previous frame f _{n -1} given by Equation 2.

s _b = s-v

If the forward motion vector is assumed to be the same as the backward motion vector, then the pixel s on the interpolated image frame f _n is in front of the pixel s _f on the next image frame f _{n +1} given by Is mapped to.

s _f = s + v

Accordingly, "s _b " and "s _f " are corresponding positions around "s".

In addition, the bidirectional motion calculator 12 uses a block motion model. If "B _{i , j} " is one block in the interpolated image frame f _n , the sum SBAD of the bidirectional absolute difference for each candidate motion vector v may be calculated as shown in Equation 4.

Through retrieval of a candidate motion vector that minimizes the sum of bidirectional motion absolute differences (SBAD), the bidirectional motion calculator 12 only interpolates the image frame from the information of the previous and next frames f _{n -1} , f _{n +1} . A motion vector of one block on (f _n ) can be calculated. If the video sequence includes simple transformed (deformed) movements, it will be demonstrated that the bidirectional motion calculator 12 yields the appropriate block motion vectors. However, if the objects have rotational or magnification movements, a reliable movement trajectory cannot be calculated. This is because many trajectories via the current block can be potential candidates with small sums of bidirectional motion absolute differences (SBADs).

In order to improve the certainty (or reliability) of the block motion operation, the bidirectional motion calculator 12 performs spatial smoothing suppression. In practice, as shown in FIG. 4, Side Match Criterion is applied to the bidirectional motion calculator 12 that measures spatial smoothness across block boundaries. According to this lateral matching scheme, the bidirectional motion calculator 12 predicts the blocks B _{i , j} and adjacent blocks B _{i -1, j} , B _{i + 1, j} , B _{i , j} along the top, bottom, left and right boundaries. The average value SMD of absolute differences between adjacent pixels of _-1 , B _{i , j + 1} ) is calculated as in Equation 5.

In Equation 5, "g _k " and "h _k " are the predicted block ( B _{i , j} ) and the adjacent blocks (B _{i -1, j} , B _{i + 1, j} , B _{i , j-1} , B _{i, j + 1} ) respectively indicate the k th pixels on the boundaries, and “N” is the number of pixels on the boundary. As shown in Equation _{1, "f n [g k} , v]" represents a pixel value to be interpolated of the prediction block (B _{i, j)} at which the motion vector is set to "v". Pixel values to be interpolated in adjacent blocks (B _{i -1, j} , B _{i + 1, j} , B _{i, j-1} , B _{i, j + 1} ), i.e., "f _n [h _k ]" are bounds It will be available for the calculation of the mean value SMD of the absolute pixel differences. Subsequently, the bidirectional motion calculator 12 performs an interpolation block _Bi to minimize the weighted substitution sum between the sum of the bidirectional motion absolute differences SBAD and the mean value SMD of the boundary pixel absolute differences, as shown in Equation (6) _{. j)} to find the best block motion vectors (v _{i, j)} for.

v _{i , j} = arg min {μSBAD [v _{i , j} , v] + (1-μ) SMD [v _{i , j} , v]}

In Equation 6, "μ" is a weighting coefficient. Since the calculation of the mean value SMD of the boundary pixel absolute difference requires pixel values of the left and upper blocks as well as the right and lower blocks, the calculation of the block motion vector is affected by the pixel values on the blocks following the interpolation block. . The minimization of the weighted substitution sum in Equation 6 is repeated for all blocks on the interpolated image frame f _n until the calculated candidate block motion vectors v _{i , j} are not changed (unchanged). Apply. Experiments have shown that when the weighting coefficient μ in equation (6) is 0.4, the bidirectional motion calculator 12 provides the best performance, and the performance is not very sensitive to the variation of “μ”. Therefore, in the bidirectional motion calculator 12, the weighting coefficient mu is fixed at 0.4. Thus, through block motion modeling and lateral matching, the bidirectional motion calculator 12 may calculate a motion vector for each block so that spatial smoothness is suppressed and reliability is improved. For example, when the previous and next image frames of FIGS. 2A and 2B are input, the bidirectional motion calculator 12 calculates block-by-block motion vectors as shown in FIG. 2C.

The object discriminator 14 converts the blocks on the interpolated image frame f _n into various parts (eg, object area, background area, and boundary area) based on the motion vectors from the bidirectional motion operator 12. Separate. Each classification portion represents roughly distinguished moving objects, including convexities with similar motion vectors. For example, when the motion vector for each block of the interpolated image frame as shown in FIG. 2C is input from the bidirectional motion calculator 12, the object discriminator 14 is divided into an object region, a background region, and a boundary region as shown in FIG. 2D. Generates an interpolated video frame. In FIG. 2D, the gray and white blocks represent the interior and boundaries of the moving object, respectively, and the black blocks depict the background. To distinguish these objects, the object classifier 14 groups similar block motion vectors from the bidirectional motion operator 12 using the k-means clustering algorithm.

Specifically, the object discriminator 14 sets, as a first step, all blocks on the interpolated image frame as one object, and sets the average motion vector of the blocks to a centralized center. In a second step, the object discriminator 14 checks whether the difference between the motion vector and the center of focus of each block is greater than the preset threshold value τ. If the difference between the motion vector of the block and the center of focus is greater than the threshold value?, The block is found to belong to a new object. In the present embodiment, the threshold value τ is set to "8" obtained through the experiment. In a third step, the object discriminator 14 assigns each block motion vector to the nearest center of focus. In the fourth step that follows, the object discriminator 14 updates each concentration center to an average of the block motion vectors included in each cluster. Finally, the moving object discriminator 14 repeats the second to fourth steps repeatedly until the focus centers do not change. Through this K-means clustering, the moving object discriminator 14 generates moving object classification information as shown in FIG. 2D using block motion vectors as shown in FIG. 2C from the bidirectional motion calculator 12.

The variable block motion compensator 16 accurately corrects the block motion vectors of the boundary region among the block motion vectors from the bidirectional motion calculator 12 based on the moving object classification information of the interpolated image frame from the object discriminator 14. Reconstruction with sub-block motion vectors reduces unwanted blurring artifacts and occlusion defects. For example, as shown in FIG. 5, a block (marked in gray) at a boundary between two objects or at a boundary between other internal blocks is called a boundary block. The motion vector of each of the boundary blocks is reconstructed by the variable block motion compensator 16 into subblock motion vectors representing the multiple motion trajectories. In other words, the variable block motion compensator 16 according to the object discrimination information from the object discriminator 14, the block motion vector corresponding to the block of the boundary portion among the block motion vectors from the bidirectional motion calculator 12. Is refined to a more precise subblock motion vector. Explaining this in detail, since the bidirectional motion calculator 12 and the object discriminator 14 use a block containing 8 × 8 pixels as the minimum unit area, the block-by-block boundary does not completely match the boundary of the authentic object. Do not. Indeed, the boundary block of the boundary portion may include multiple objects of different movement, as in FIG. 6. To represent such complex motion, variable block motion compensator 16 reconstructs the boundary blocks. The variable block motion compensator 16 is more effective than fixed size block motion compensating means in exhibiting irregular motion. Smaller subblocks are used to represent complex movements near object boundaries, while larger blocks are used to represent homogeneous movement regions.

To this end, the variable block motion compensator 16 divides an 8x8 boundary block into subblocks containing 4x4 or 2x2 pixels, as shown in FIG. 6. Perform variable-size block motion compensation of. _{Assume that} "B _l ⁽ⁿ⁾ " represents the l-th subblock of size 2 ⁿ x 2 ⁿ and "v _l ⁽ⁿ⁾ " represents the motion vector of the sub block B _l ⁽ⁿ⁾ . Then, the variable block motion compensator 16 first initializes "n" to 3 (that is, n = 3). In a second step, the variable block motion compensator 16 converts the current block of "2 ⁿ x 2 ⁿ " into four sub blocks of "2 ^n-1 x 2 ^n-1 ", as shown in equation (7). Divide.

B _l ⁿ → {B ₀ ^(n-1) , B ₁ ^(n-1) , B ₂ ^(n-1) , B ₃ ^(n-1) }

In this case, the variable block motion compensator 16 measures the sum of the bidirectional motion absolute differences (SBAD) for each candidate motion vector as shown in Equation 4, and then measures each subblock B _m ^(n-1) . Find the motion vector v ₀ ^(n-1) . Article as step 3, a variable block motion compensator (16) is a sum of the sub-motion vector absolute differences for all of the sub-block ^{_{"SABD [B m (n-}} 1), v m (n-1)]" is 2 ⁿ × 2 block motion vectors the sum of absolute differences ^{_{"SABD [B ℓ (n)}} , v ℓ (n)]" to the source block of ⁿ is less than the quarter, it is determined that an acceptable sub-divided. In other words, if 0 ≤ m ≤ 3, if Equation 8 holds,

SABD [B _m ^(n-1) , v _m ^(n-1) ] <(1/4) · SABD [B ₁ ⁽ⁿ⁾ , v ₁ ⁽ⁿ⁾ ], when "0≤m≤3"

The variable block motion compensator 16 allows subdivision to reduce " n " to n-1. Otherwise, variable block motion compensator 16 retains the original block and ends the procedure. As a final step, the variable block motion compensator 16 checks whether "n" is one. Here, if "n" is 1, the variable block motion compensator 16 ends the procedure, while if "n" is not 1, the variable block motion compensator 16 returns to the second step. As such, the variable block motion compensator 16 reconstructs the block motion vector at the boundary portion of the object into more precise sub-motion vectors, thereby reducing the defect of the fixed-size block motion compensation scheme. In fact, when block-by-block motion vectors such as FIG. 2C or FIG. 5 are calculated in the bidirectional motion calculator 12, the variable block motion compensator 16 responds to the moving object classification information from the object discriminator 14. It generates variable block motion vectors such as FIG. 2E or FIG. 6, which are reconstructed into more precise sub-block motion vectors.

AOBMC type The pixel interpolator 18 uses adaptive window coefficients so that the block motion vectors on the background and object regions are adaptively compensated, and the block motion vectors and the previous and next image frames f _{n -1} , f. _{n +1} ) to interpolate pixels on the interpolated image frame f _n . The AOBMC type pixel interpolator 18, unlike the superimposed block motion compensated pixel interpolator of the related art, which applies only the motion vector of the block to interpolation of the fixtures of each block, uses a weighted-substituted adaptive window to block movement. Apply a vector to a larger set of pixels. For convenience of explanation, it is assumed that "s" is a pixel located in the lower right quadrant of the (i, j) th block (B _{i , j} ). The AOBMC type pixel interpolator 18 includes the pixels f _n [s] of the blocks B _{i , j} , the block itself, the lower blocks B _{i +1, j} , the right blocks B _{i , j + 1} ), and the motion vectors of the lower right block B _{i + 1, j + 1} . _{Assume that} "v _{i + p, j + q} " represents the motion vector of the block (B _{i + p, j + q} ) and "w _{p, q} " represents the corresponding weighted substitution window coefficient. Then, the AOBMC type pixel interpolator 18 may calculate the prediction pixel f _n [s] as shown in Equation (9).

The weighted substitution window coefficient w _{p , q} [s] is determined by the relative position of the pixel s in the block _{Bi , j} . Indeed, smoothing functions, such as the raised cosine window from (11), are often used for the weighted substitution window coefficients w _{p , q} [·]. The method of predicting pixels of other quadrants can be derived more easily. For example, for the upper left quadrant of a block, the motion vectors of the block itself, the upper block, the left block, and the upper left block are used.

In general, if motion activity is low, overlapping block motion compensation can effectively attenuate occlusion artifacts to provide good visual quality. Conversely, if adjacent blocks have substantially different movements, overlapping block motion compensation may result in blurring or excessive smoothing artifacts. Suppose two blocks are adjacent to each other, one of which is part of a fast moving object while the other is in the background. In this case, the motion vector of the moving block is not affected by the pixels of the background block. Unfortunately, in the overlapping block motion compensation of the related art, the weighted substitution coefficients are determined only by the relative distance (or spacing) of the pixels in the block, so that the boundary pixels of the background block are sufficiently affected by the motion vector of the moving block. Receive.

To overcome this drawback, the AOBMC type pixel interpolator 18 adaptively controls weighted displacement window coefficients according to the reliability of adjacent motion vectors. 7 compares-explains the changing states of the weighted displacement window coefficients according to pixel position by the superimposed block motion compensated pixel interpolator and AOBMC type pixel interpolator 18 of the related art. In FIG. 7, the dotted line waveform illustrates the state of change of the weighted displacement window coefficient according to the position of the pixel by the superimposed motion block compensated pixel interpolator of the related art, and the solid line waveform is transferred to the AOBMC type pixel interpolator 18. The change state of the weighted substitution window coefficient according to the pixel position will be described. 7, the blocks each pixel (s) right half portion of (B _{i, j),} block weighted motion by substituted sum of the two pixel values specified by the motion vector of (B _{i, j)} - Is compensated. In the superimposed block motion compensated pixel interpolator of the related art, the weighted displacement window coefficients (w _{i , j} [s] and w _{i + 1, j} [s]) are the pixels (s) in the block (B _{i , j} ). Is determined by its relative position. In contrast, the AOBMC type pixel interpolator 18 adaptively changes the weighted substitution window coefficients according to the reliability of the adjacent motion vector. As an example, assume that the motion vector of the block _{Bi + 1, j} is not reliable enough to be used for the prediction of the block _{Bi , j} . In this case, the Fig., As in 7, corresponding weighting substituted window coefficients _{(w i + 1, j [} s]) is different weighting substituted window coefficients on the other hand is reduced to _{"w 'i + 1, j} [s]" , which (w _{i , j} [s]) increases to "w ' _{i , j} [s]".

In order to adaptively change the weighted substitution window coefficients, the AOBMC type pixel interpolator 18 calculates the reliability of the neighboring block motion vectors v _{i + p, j + q} for prediction of the current blocks _{Bi and j} . Calculate as shown in Equation 10.

If the block motion vector v _{i , j} is chosen to minimize the weighted substitution sum in Equation 6, including the value of the sum of bidirectional motion absolute differences (SBAD), then in most cases the The sum SBAD [B _{i , j} , v _{i , j} ] is equal to or less than the sum of the absolute differences in bidirectional motion of adjacent blocks (SBAD [B _{i , j} , v _{i + p, j + q} ]) Becomes "0" and "1" and one of the values therebetween, that is, "0 ≤ Φ _{Bi , j} [v _{i + p, j + q} ] ≥ 1". As the reliability Φ B _{i, j} [v _{i + p, j + q} ] increases, the adjacent motion vector v _{i + p, j + q} determines the current (or predictive) block B _{i , j} . It can be used more reliably for motion-compensation. As a result, using the reliability information, the weighted substitution window coefficients w _{p and q} [s] in Equation 10 are transformed into the form of Equation 11.

Substituted changed weighting window coefficients is proportional to the reliability _{(ΦB i, j [v i} + p, j + q]). The denominator in (11) is needed to normalize the weighted substitution coefficients as in (9a). FIG. 8 illustrates a method of changing a rising cosine window in FIG. 7 according to reliability of a motion vector of an adjacent block. As in FIG. 8, if the motion vector of the adjacent block has a reliability of "1", the AOBMC type pixel interpolator 18 does not change the rising cosine window. On the contrary, if the motion vector of the adjacent block has a reliability of "0", the AOBMC type pixel interpolator 18 does not use the rising cosine coefficient for the current block, but the weighted substitution window is used for the overlapping block motion of the related art. As with the compensated pixel interpolator, they are at right angles (or rectangles). In this case, the AOBMC type pixel interpolator 16 calculates each pixel f _n [s] of the interpolation block _{Bi , j} as shown in equation (12).

f _n [s, v _ℓ ⁽ⁿ⁾ ] = (f _{n -1} [sv _ℓ ⁽ⁿ⁾ ] + f _{n +1} [s + v _ℓ ⁽ⁿ⁾ ]) / 2, where s∈B _ℓ ⁽ⁿ⁾ Occation

Therefore, the adaptive overlapping block motion compensator according to the present embodiment can control the window shape to solve the disadvantage of the overlapping block motion compensator of the related art. As such, the AOBMC type pixel interpolator 18 applies an interpolated image frame to the pixels on the previous and next image frames and the block motion vectors by applying or not applying an adaptive change of the weighted substitution window coefficient according to the reliability of the motion vector of the adjacent block. Interpolate the pixels of. For example, previous and next image frames, such as FIGS. 2A and 2B, and block-by-block motion vectors, such as FIG. 2E, are input from the first delayer 10, the input line 11, and the variable block motion compensator 16. In this case, the AOBMC type pixel interpolator 18 generates an interpolated image frame as shown in FIG. 2F.

In the interpolated image frame output from the AOBMC type pixel interpolator 18, the occlusion artifacts are significantly reduced and the sharp edges of the object are preserved. In other words, the frame interpolation unit according to the exemplary embodiment disclosed herein interpolates a video frame of better quality. This is clearly shown through the contrast with the interpolated video frame by the frame interpolation algorithm of the related art.

9A-9C are interpolated image frames from an image frame column of a football with fast and complex movements, and FIGS. 10A-10C are interpolated from an image frame column of architectural tens having modest (ie, simple) movements. 11A-11C are image frames interpolated from a series of image frames of a ping pong with fast moving objects and scene transitions (changes), and FIGS. 12A-12C are views of a vehicle comprising multiple objects of divergent movements. Image frames interpolated from an image frame column. 9A through 12A are interpolated image frames generated by a 3D inductive motion compensated frame interpolation algorithm, FIGS. 9B through 12B are interpolated image frames generated by a motion compensated frame interpolation algorithm of Zhai et al, and 9C to 12C are interpolated image frames generated by an AOBMC type frame interpolator according to an exemplary embodiment of the present specification.

For interpolation of the interpolated image frames of FIGS. 9 to 12, the block size is set to a size including 8 × 8 pixels and the search range is set to 16 pixels in both horizontal and vertical directions. will be. In addition, by the variable-size block motion compensator 16, each of the boundary blocks has been divided into multiple 4x4 or multiple 2x2 subblocks. In addition, the image frames of FIGS. 9 to 12 were interpolated with the frame rate set to 60 fps, which is twice the original frame rate of 30 frames per second. In the interpolated image frames of FIGS. 9C to 12C interpolated by the frame interpolator according to an exemplary embodiment of the present disclosure, the interpolated image frames of FIGS. 9A to 12A by a three-dimensional inductive motion compensation frame interpolation algorithm are, of course, Zhai et al. As compared to the interpolated image frames of FIGS. 9B to 12B by al's frame interpolation algorithm, the occlusion artifacts are significantly reduced and the edges of the object are sharply preserved. This quality improvement of the image can be easily observed around the objects on the interpolated image frame of FIGS. 9 to 12, i.e. the bodies of football players, the face of an architectural leader, the arms of a table tennis player, and the rolling ball.

In addition, the AOBMC type frame interpolation unit according to an embodiment of the present disclosure exhibits an improved signal-to-noise ratio as compared to the 3D inductive motion compensation frame interpolation algorithm and Zhai et al's motion compensation frame interpolation algorithm. Although the nth interpolated picture frame f _n is interpolated from the previous (n-1st) picture frame and the next (n + 1th) picture frame, the signal-to-noise ratio (PSNR) is interpolated with the original nth frame. Operates between nth frames. If "f _n " and "f ' _n " represent the original video frame and the interpolated video frame, respectively, the signal-to-noise ratio (PSNR) is defined as in Equation (13).

In Equation 13, "W x H" is the spatial resolution of the image string. 13A-13D illustrate the characteristics of the signal-to-noise ratio for interpolated image frames interpolated from 100 frames in the “football”, “architecture”, “ping pong”, and “car” image columns. 13A to 13D, "PIC" indicates a conversion state of a signal-to-noise ratio corresponding to 50 interpolated image frames interpolated by an AOBMC type frame interpolator according to an embodiment of the present specification, and "RA1" indicates Represents the conversion state of the signal-to-noise ratio according to the 50 interpolated image frames interpolated by the 3D inductive motion compensation frame interpolation algorithm. Finally, "RA2" represents 50 interpolated images interpolated by Zhai et al's frame interpolation algorithm. The conversion state of the signal-to-noise ratio according to the frame is shown. As shown in FIGS. 13A to 13D, the AOBMC type frame interpolator according to an embodiment of the present specification has a better signal-to-noise ratio of about 2 dB, compared to a 3D inductive motion compensation frame interpolation algorithm and Zhai et al's frame interpolation algorithm. Providing ratio performance.

The apparatus for upconverting the frame rate of FIG. 1 increases the rate at which the delayed image frame f _{n -1} from the first delayer 10 and the interpolated image frame f _n from the AOBMC type pixel interpolator 18 are input. And a combination unit 20. The rate up and combiner 20 up-converts the delayed image frame f _{n -1} and the interpolated image frame f _n of the first frame frequency to a second frame frequency higher than the first frame frequency. The second frame frequency may be set to correspond to 1.5 times or 2 times the first frame frequency. For example, when the first frame frequency is 60 Hz, the second frame frequency is 90 Hz or 120 Hz. In addition, the speed up and combiner 20 inserts the interpolated image frame f _n between the delayed image frames f _{n -1} and f _{n + 1} to speed up the converted image frame sequence (ie, the second frame). A video frame sequence fru of frame frequency is generated. The speed up-converted image frames fru generated by the speed up and combination unit 20 are sequentially output through the output line 13. When the second frame frequency is set to 1.5 times the first frame frequency, the speed increasing and combining unit 20 inserts one interpolated image frame f _n every two delayed image frames. On the contrary, if the second frame frequency is set to twice the first frame frequency, the speed increasing and combining unit 20 alternately arranges the delayed image frame and the interpolated image frame. Furthermore, the speed increasing and combining unit 20 may perform the operation of increasing the speed of the image frame and the combining operation of the image frame sequentially or simultaneously. In order to sequentially or simultaneously perform the speed up operation and the combination operation, the speed up and combination unit 20 includes a memory having a capacity of at least four frames.

The output line 13 may be connected to a hold display device such as a liquid crystal display, an electroluminescent display, and a plasma display. Since the hold display device connected to the output line 13 is driven by the frame rate up-converted image frames fru, the outline of the object is sharply preserved while minimizing the occurrence of blurring and occlusion artifacts. High quality images can be provided.

In addition, the frame rate up-conversion apparatus of the embodiment may further include a second delay unit 22 connected between the first delay unit 10 and the speed up unit and the combination unit 20. The second delayer 22 is used for data processing time of the frame interpolator (i.e., in the bidirectional motion calculator 12, the object discriminator 14, the variable block motion compensator 16, and the AOBMC type pixel interpolator 18). Secondly delays the image frames f _{n -1} , f _{n +1} to be supplied to the speed increasing and combining unit 22 from the first delay unit 10 for a period corresponding to . By the delay operation of the second delayer 22, the speed up and combination unit 20 is delayed image frame (f _{n -1} , f _{n +1} ) and AOBMC type output from the first delayer 10. The interpolation image frame f _n output from the pixel interpolator 18 may be simultaneously input.

As described above, the frame interpolation apparatus of the embodiment disclosed herein and the frame rate up-conversion apparatus of the embodiment including the same effectively predict motion vectors without generating hole and overlapping artifacts through bidirectional motion calculation. . Further, the frame interpolation apparatus of the embodiment and the frame rate up-conversion apparatus of the embodiment including the same adaptively control the weighted substitution window coefficients based on the reliability of the adjacent motion vectors, and the adaptive weighted substitution window coefficients are optional. By performing pixel interpolation applied, the excessive smoothing is suppressed. Accordingly, the frame interpolation apparatus of the embodiment can provide a high quality interpolation image frame in which the outline of an object is sharply preserved while minimizing occlusion artifacts. In addition, the apparatus for up-converting the frame rate of the embodiment may up-convert the image frame sequence to the frame rate while minimizing deterioration of the occlusion artifact and the outline of the object. In addition, the frame interpolation apparatus of the embodiment and the frame rate up-conversion apparatus of the embodiment including the same may improve the signal-to-noise ratio of the image frame by approximately 2 dB.

As described above, embodiments of the present invention will be described with reference to FIGS. 1 to 13 by limiting the frame interpolation apparatus and the frame rate up-conversion apparatus using only two frames (ie, previous and next frames) to interpolate the intermediate frame. However, it will be apparent to those skilled in the art that various modifications, changes, and equivalent embodiments may be made without departing from the spirit and scope disclosed by the embodiments. . For example, to improve the quality of an interpolated frame, one of ordinary skill in the art will readily recognize that at least three adjacent frames may be used. Accordingly, the spirit and scope disclosed in the embodiments should not be limited to the description of the embodiments, but should be set by the matters set forth in the appended claims.

1 is a block diagram schematically illustrating an apparatus for upconverting a frame rate according to an embodiment of the present invention.

2A to 2F are diagrams illustrating image frames indicating input and output results of respective parts of FIG. 1.

FIG. 3 is a diagram illustrating a bidirectional motion calculation process performed by the bidirectional motion calculator of FIG. 1.

FIG. 4 is a diagram illustrating a lateral matching process performed by the bidirectional motion calculator of FIG. 1.

FIG. 5 is a diagram illustrating in detail block motion vectors clustered by region by the object discriminator of FIG. 1.

FIG. 6 is a diagram illustrating a state in which boundary block motion vectors are precisely reconstructed by the variable block motion compensator of FIG. 1.

FIG. 7 is a diagram for comparing and explaining changes of weighted displacement window coefficients according to pixel positions by an overlapping block motion compensated pixel interpolator of the related art and an AOBMC type pixel interpolator of FIG. 1.

FIG. 8 is a diagram for describing a method of changing the rising cosine window according to the reliability of a motion vector of an adjacent block.

9A to 9C illustrate footframe image frames interpolated by the frame interpolation method of the related art and the frame interpolation unit of FIG. 1.

10A to 10C illustrate image frames of an architectural dozen interpolated by the frame interpolation method of the related art and the frame interpolation unit of FIG. 1.

11A to 11C illustrate table tennis image frames interpolated by the frame interpolation method of the related art and the frame interpolation unit of FIG. 1.

12A to 12C illustrate vehicle image frames interpolated by the frame interpolation scheme of the related art and the frame interpolator of FIG. 1.

13A to 13D illustrate contrast-description characteristics of PSNRs for the image frame columns of FIGS. 9 to 12 interpolated by the related art of interpolating the image frames of FIGS. 9 to 12 and the frame interpolation unit of FIG. 1.

`` Explanation of symbols for main parts of drawings ''

10: first delay unit 12: bidirectional motion calculator

14 object discriminator 16 variable block motion compensator

18: AOBMC type pixel interpolator 20: second delay

22: speed up and combiner

Claims (12)

A motion calculator for calculating a motion vector for each block of the interpolated video frame by performing bidirectional comparison using pixel information of a previous video frame and a next video frame; And Based on the reliability of the motion vector of the adjacent block and the block to be interpolated on the interpolated image frame, the previous and next images to which the adaptive weighted substitution window coefficient is selectively applied as well as adaptively adjusted. And an adaptive superimposed motion compensated pixel interpolator for interpolating pixels on the interpolated image frame by performing calculation of pixel information of the frame and the motion vector of the interpolation block. The method of claim 1, An object discriminator for classifying a boundary area between objects on the interpolated image frame based on the block-by-block motion vectors from the bidirectional motion calculator; And The sub-block motion vectors and the block motion vector are reconstructed into sub-block motion vectors by reconstructing motion vectors of the boundary block included in the boundary area distinguished by the object identifier among the block motion vectors from the bidirectional motion calculator. And a variable block motion compensator for supplying variable block motion vectors including the adaptive block motion vectors to the adaptive overlapping block motion compensated pixel interpolator. The method of claim 1, And the adaptive overlapping block motion compensated pixel interpolator does not apply the adaptively adjusted weighted substitution window coefficient to pixel interpolation when the reliability of the motion vector of the neighboring block is "0". Calculating a motion vector for each block of the interpolated image frame through bidirectional comparison using pixel information of a previous image frame and a next image frame; Calculating a reliability of a motion vector of a block to be interpolated and an adjacent block on the interpolated image frame; Adjusting a weighted substitution window coefficient based on the reliability of the motion vector of the adjacent block; And According to the reliability of the neighboring block, the pixel information of the previous and next image frame frame and the motion vector of the interpolation block are applied by selectively applying the adjusted weighted substitution window coefficient to interpolate pixels on the interpolated image frame. Frame interpolation method comprising the step of. The method of claim 4, wherein the motion vector calculation step comprises: Classifying a boundary region between objects on the interpolated image frame based on the block-by-block motion vectors; And And reconstructing the motion vector of the boundary block included in the boundary region among the block motion vectors into sub-block motion vectors. The method of claim 5, wherein And the pixel interpolation step does not apply the adjusted weighted substitution window coefficient to pixel interpolation when the reliability of the motion vector of the adjacent block is "0". A delayer for delaying an image frame of a first frame frequency from an input line for a period of the first frequency frame; A motion calculator for calculating a motion vector for each block of the interpolated image frame by bidirectional comparison using pixel information of the image frame from the delayer and the image frame from the input line; The input line and the adaptively adjusted weighted substitution window coefficients are adaptively applied based on the reliability of the motion vector of the adjacent block and the block to be interpolated on the interpolated image frame. An adaptive superimposed motion compensated pixel interpolator for performing an operation on the pixel information of the image frames from the delayer and the motion vector of the interpolation block to interpolate pixels on the interpolated image frame; And Between the delayed image frame from the delayer and the interpolated image plane from the adaptive overlapping block motion compensated pixel interpolator to a second frame frequency higher than the first frame frequency and upstream of the delayed image plane And a rate up and combiner configured to insert-combine the interpolated image frame of which the frame frequency is increased. The method of claim 7, wherein An object discriminator for classifying a boundary area between objects on the interpolated image frame based on the block-by-block motion vectors from the bidirectional motion calculator; And The sub-block motion vectors and the block motion vector are reconstructed into sub-block motion vectors by reconstructing motion vectors of the boundary block included in the boundary area distinguished by the object identifier among the block motion vectors from the bidirectional motion calculator. And a variable block motion compensator for supplying the variable block motion vectors including the variable block motion vectors to the adaptive overlapping block motion compensated pixel interpolator. The method of claim 8, And the second frame frequency is set to any one of 1.5 times and 2 times the first frame frequency. Delaying an image frame of a first frame frequency on an input line for a period corresponding to a period of the first frame frequency; Calculating a motion vector for each block of the interpolated image frame through bidirectional comparison using pixel information of the image frame and the delayed image frame on the input line; Calculating a reliability of a motion vector of a block to be interpolated and an adjacent block on the interpolated image frame; Adjusting a weighted substitution window coefficient based on the reliability of the motion vector of the adjacent block; According to the reliability of the neighboring block, the pixel information of the previous and next image frame frame and the motion vector of the interpolation block are applied by selectively applying the adjusted weighted substitution window coefficient to interpolate pixels on the interpolated image frame. Doing; And Interpolating the delayed image frame and the interpolated image plane to a second frame frequency higher than the first frame frequency and interpolating the frame frequency upward interpolated image frame between the frame frequency upward delayed image planes; Frame rate up-conversion method, characterized in that. The method of claim 10, wherein the motion vector operation step, Classifying a boundary region between objects on the interpolated image frame based on the block-by-block motion vectors; And And reconstructing a motion vector of a boundary block included in the boundary region among the block motion vectors into sub-block motion vectors. The method of claim 5, wherein And the pixel interpolation step does not apply the adjusted weighted substitution window coefficient to pixel interpolation when the reliability of the motion vector of the adjacent block is "0".

Images