KR20200024572A - Apparatus and method for motion compensated frame interpolation suitable for both linear and nolinear motion - Google Patents

Abstract

Disclosed are a movement compensation frame interpolation device suitable for both linear and nonlinear movement and a method thereof. According to an embodiment of the present invention, the interpolation device comprises: an estimator estimating a first movement vector by using a current frame and a previous frame and estimating a second movement vector by further using a frame consecutive to any one of the current frame and the previous frame; and a generator generating an intermediate frame between the current frame and the previous frame by using any one of the first motion vector and the second motion vector, wherein the first motion vector and the second motion vector are estimated for each block unit of a frame.

Description

Apparatus and method for motion compensation frame interpolation suitable for both linear and non-linear motion {APPARATUS AND METHOD FOR MOTION COMPENSATED FRAME INTERPOLATION SUITABLE FOR BOTH LINEAR AND NOLINEAR MOTION}

The embodiments below relate to an apparatus and method for motion compensation frame interpolation suitable for both linear and nonlinear motion.

When there is a bandwidth limitation in a communication system, the video frame rate is lowered to meet the limitation. In the case of a liquid crystal display (LCD), temporal distortion such as motion blur occurs when the frame rate of the video is low. These problems can be overcome by improving the frame rate of the video at the decoder.

Motion Compensated Frame Interpolation (MCFI) is widely used to improve the frame rate of video at the decoder. MCFI is a technique that improves the frame rate by estimating the motion of an object on consecutive video frames and generating and inserting an intermediate frame.

In order to improve the frame rate, it is important to accurately estimate the motion of an object on a video frame.

Embodiments may provide a technique for accurately estimating the motion of an object on a video frame by selecting motion vectors estimated by a motion estimation method that is more suitable for each motion when the object exhibits linear or nonlinear motion within the frame. Can be.

In addition, embodiments may generate an intermediate frame using a correctly estimated motion vector, thereby improving the frame rate of the video and providing a motion compensation frame interpolation technique suitable for both linear and nonlinear motion.

According to an exemplary embodiment, an interpolation apparatus estimates a first motion vector using a current frame and a previous frame, and further estimates a second motion vector using a frame continuous to any one of the current frame and the previous frame. An estimator and a generator for generating an intermediate frame between the current frame and the previous frame by using any one of the first motion vector and the second motion vector; and the first motion vector and the second motion vector. Is estimated for each block unit of a frame.

The estimator estimates a first forward motion vector through forward motion estimation based on the current frame using the previous frame and the current frame, and refers to the previous frame using the current frame and the previous frame. The first backward motion vector may be estimated through backward motion estimation, and the first motion vector may be estimated using any one of the first forward motion vector and the first backward motion vector.

The estimator calculates a sum of absolute difference (SAD) value of the first forward motion vector and the first backward motion vector, and calculates a sum of absolute difference (SAD) value of the first forward motion vector and the first backward direction The first motion vector may be estimated using a motion vector having a smaller SAD value among the SAD values of the motion vector.

The estimator is bidirectional based on an intermediate frame between the current frame and the previous frame using a motion vector having a smaller SAD value among the SAD value of the first forward motion vector and the SAD value of the first backward motion vector. The first motion vector may be estimated through the motion estimation method.

The estimator estimates a second forward motion vector through forward motion estimation based on the previous frame using the current frame, the previous frame, and the previous frame of the previous frame, and estimates the current frame, the previous frame, and A second backward motion vector is estimated through backward motion estimation based on the subsequent frame using a later frame of the current frame, and a motion vector of any one of the second forward motion vector and the second backward motion vector is calculated. The second motion vector may be estimated by selecting.

The estimator calculates a sum of absolute difference (SAD) value of the second forward motion vector and the second reverse motion vector, and calculates a sum of absolute difference (SAD) value of the second forward motion vector and the second reverse motion vector. The second motion vector may be estimated using a motion vector having a smaller SAD value among the SAD values of the motion vector.

The generator may include a selector configured to output a motion vector field (MVF) by selecting one of the first motion vector and the second motion vector for each block unit of a frame, and corrected by removing an error from the MVF. And a filter for outputting an MVF, an intermediate frame based on the modified MVF, and a modifier for generating an intermediate frame between the current frame and the previous frame using the modified MVF.

The selector may select the first motion vector when the motion of the object in the frame is nonlinear and select the second motion vector when the motion of the object in the frame is linear.

The modifier detects a distortion region of an intermediate frame based on the modified MVF, modifies a motion vector of the modified MVF corresponding to the distortion region, generates a remodified MVF, and uses the remodified MVF. An intermediate frame may be generated between a current frame and the previous frame.

The modifier may detect a distortion region through a convolutional neural network (CNN).

According to an exemplary embodiment, an interpolation method includes estimating a first motion vector using a current frame and a previous frame, and further extracts a second motion vector using a frame continuous to any one of the current frame and the previous frame. Estimating and generating an intermediate frame between the current frame and the previous frame by using any one of the first and second motion vectors. The second motion vector is estimated for each block unit of the frame.

The estimating of the first motion vector may include estimating a first forward motion vector through forward motion estimation based on the current frame using the previous frame and the current frame, and the current frame and the previous motion vector. Estimating a first backward motion vector by using backward motion estimation based on the previous frame, and using one of the first forward motion vector and the first backward motion vector. Estimating the first motion vector.

The estimating of the first motion vector using the motion vector may include calculating a sum of absolute difference (SAD) value of the first forward motion vector and the first backward motion vector, The method may include estimating the first motion vector using a motion vector having a smaller SAD value from a sum of absolute difference (SAD) value of the first forward motion vector and a SAD value of the first backward motion vector.

The estimating of the first motion vector using the motion vector having the smaller SAD value may include: a motion having a smaller SAD value among the SAD value of the first forward motion vector and the SAD value of the first backward motion vector; The method may include estimating a first motion vector through a bidirectional motion estimation method based on an intermediate frame between the current frame and the previous frame using a vector.

The estimating of the second motion vector may include estimating a second forward motion vector through forward motion estimation based on the previous frame using the current frame, the previous frame, and a previous frame of the previous frame; Estimating a second backward motion vector by using backward motion estimation based on the next frame using the current frame, the previous frame, and subsequent frames of the current frame, and the second forward motion vector and the first frame. Estimating the second motion vector by selecting one of the two backward motion vectors.

The estimating of the second motion vector by selecting the one motion vector may include calculating a sum of absolute difference (SAD) value between the second forward motion vector and the second backward motion vector, And estimating the second motion vector using a motion vector having a smaller SAD value from a sum of absolute difference (SAD) value of the two forward motion vectors and a SAD value of the second backward motion vector.

The generating may include selecting a motion vector of the first motion vector and the second motion vector for each block unit of a frame and outputting a motion vector field (MVF), and a filter may detect an error in the MVF. Removing and outputting a modified MVF, and generating an intermediate frame based on the modified MVF and an intermediate frame between the current frame and the previous frame using the modified MVF.

The outputting may include selecting the first motion vector when the motion of the object in the frame is nonlinear, and selecting the second motion vector when the motion of the object in the frame is nonlinear. It may include.

The generating may include generating an intermediate frame based on the modified MVF, detecting a distortion region of the intermediate frame based on the modified MVF, and moving the motion vector of the modified MVF corresponding to the distortion region. And generating a re-modified MVF, and generating an intermediate frame between the current frame and the previous frame by using the re-modified MVF.

The detecting may include detecting a distortion region of an intermediate frame based on the modified MVF using a convolutional neural network (CNN).

1 is a diagram illustrating a unidirectional motion estimation method.
2 is a diagram illustrating a bidirectional motion estimation method.
3 is an illustration of an estimated motion vector field.
4 is a schematic block diagram of an interpolation apparatus according to an embodiment.
FIG. 5 is a detailed block diagram of the estimator shown in FIG. 4.
6 is a diagram illustrating a forward motion estimation method among motion estimation methods using two frames.
7 is a diagram illustrating a backward motion estimation method among motion estimation methods using two frames.
8 is a diagram illustrating a bidirectional motion estimation method among motion estimation methods using two frames.
9 is a diagram illustrating a motion estimation method using three frames.
FIG. 10 is a detailed block diagram of the generator shown in FIG. 4.
11 is a diagram illustrating a distortion area detected by a crystal.
12 is a diagram illustrating a distortion elimination process according to a repetition of modification of a corrector.
13 is a flowchart illustrating an interpolation method, according to an exemplary embodiment.
14 is a diagram illustrating average PSNR and SSIM measurement results for each sequence.
15 is a diagram illustrating a result of a mobile sequence.
16 shows the results of a Soccer sequence.
17 shows the result of a Stefan sequence.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments so that the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, and substitutes for the embodiments are included in the scope of rights.

The terminology used herein is for the purpose of description and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The terms are intended only to distinguish one component from another, for example, without departing from the scope of rights according to the concepts of the embodiment, the first component may be named a second component, and similarly The second component may also be referred to as a first component.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

In addition, in the description with reference to the accompanying drawings, the same components will be given the same reference numerals regardless of the reference numerals and duplicate description thereof will be omitted. In the following description of the embodiment, if it is determined that the detailed description of the related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 is a diagram illustrating a unidirectional motion estimation method, FIG. 2 is a diagram illustrating a bidirectional motion estimation method, and FIG. 3 is an exemplary diagram of an estimated motion vector field.

1 to 3, a motion compensated frame interpolation method (MCFI) refers to a technique of generating an interpolated frame between successive frames to improve the frame rate of video content. . MCFI can be divided into motion estimation, motion vector correction, and intermediate frame generation.

The MCFI uses a motion estimation method to estimate the motion of an object between successive frames. In this case, a block matching algorithm (BMA) is mainly used in consideration of appropriate time complexity and performance. The block matching algorithm divides an input frame into blocks and estimates a motion vector for each block.

The motion estimation method is divided into a uni-directional motion estimation method and a bi-directional motion estimation method according to which frame the reference is taken.

The unidirectional motion estimation method may be as shown in FIG. 1. In the unidirectional motion estimation method, when an interpolated frame (t-1 / 2) is generated between a previous frame (t-1) and a current frame (t), the motion is based on the current frame. Estimate the vector. The motion vector is estimated using Equation 1.

는 단방향 움직임 벡터,

는 이전 프레임

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here

Is a one-way motion vector,

Frame the old

Means the current frame. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

The unidirectional motion estimation method can estimate a motion vector with high accuracy, but inevitably, an overlap region and a hole region are generated when generating an intermediate frame.

The bidirectional motion estimation method may be as shown in FIG. 2. When an interpolated frame t-1 / 2 is generated between a previous frame t-1 and a current frame t, a motion vector is estimated based on the intermediate frame. The motion vector is estimated using equation (2).

는 양방향 움직임 벡터,

는 이전프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here

Is a bidirectional motion vector,

Is the previous frame,

Means the current frame. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

Since the bidirectional motion estimation method obtains motion vectors based on the intermediate frame t-1 / 2, an overlap region and a hole region do not occur. In a general bidirectional motion estimation method, a block of an intermediate frame t-1 / 2 serves as a reference point when estimating a motion between a previous frame t-1 and a current frame t. However, since the block of the intermediate frame (t-1 / 2) does not have pixel information yet, it serves as a positional reference point and does not help to find an accurate motion vector. As a result, the bidirectional motion estimation method often estimates a wrong motion vector, and the accuracy of motion vector estimation is inferior.

A motion vector is estimated for all blocks of a frame through a motion estimation process using a unidirectional motion estimation method and a bidirectional motion estimation method. The motion vectors in block units obtained by the motion estimation method constitute a motion vector field (MVF) as shown in FIG. 3.

The exact motion vector is not always estimated in the motion estimation process. Therefore, there are wrongly estimated motion vectors on the MVF, and they must be corrected. Most MCFI methods apply mean filtering, median filtering, and the like to the estimated motion vectors. Filtering the estimated motion vector smoothes the motion vector through the neighboring motion vectors, thereby obtaining a coherent MVF with reduced error.

However, some motion vectors may not be corrected even if filtering is applied. Incorrectly estimated motion vectors that exist even after filtering cause distortion in the region when the intermediate frame is generated. Incorrectly estimated motion vectors that exist even after filtering are not removed well by additional filtering. Incorrectly estimated motion vectors that exist even after filtering over-smoothing the correctly estimated motion vectors when additional filtering is performed.

In the middle frame generation step, which is the last step of the MCFI, the intermediate frame is generated through the modified vectors through filtering. In this case, the intermediate frame is generated using pixel values of the previous frame and the current frame corresponding to the position indicated by the modified motion vector. The intermediate frame is generated using equation (3).

는 중간프레임,

과

는 각각 이전 프레임과 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들을 의미한다.이처럼, MCFI는 비디오 프레임 상의 물체의 움직임을 추정하고 이를 반영하여 중간 프레임을 생성, 삽입함으로써 프레임률을 향상시킨다. 따라서, 프레임 상의 물체의 움직임을 정확히 추정하는 것이 중요하다.here

Is the middle frame,

and

Denotes the previous frame and the current frame, respectively. Also

In this way, MCFI estimates the motion of an object on a video frame and reflects it to generate and insert an intermediate frame to improve the frame rate. Therefore, it is important to accurately estimate the motion of the object on the frame.

As described above, in the unidirectional motion estimation method, an overlap region and a hole region inevitably occur when generating an intermediate frame. In addition, since the bidirectional motion estimation method finds a motion vector based on an intermediate frame having no pixel information, an incorrectly estimated motion vector exists, and thus, the accuracy of motion vector estimation is reduced. In the filtering process, incorrectly estimated motion vectors that exist even after filtering over smooth the correctly estimated motion vectors.

Embodiments may estimate a more accurate motion vector and generate an intermediate frame of improved quality using the estimated motion vector. Hereinafter, this will be described in detail with reference to FIGS. 4 to 17.

4 is a schematic block diagram of an interpolation apparatus according to an embodiment.

Referring to FIG. 4, the interpolation apparatus 100 includes an estimator 200 and a generator 300.

The estimator 200 may estimate the motion vector using two or more motion estimation methods using one or more frames. The motion estimation method may use a motion estimation method using two frames and a motion estimation method using three frames.

The estimator 200 may estimate the first motion vector using two frames and estimate the second motion vector using three frames. The first motion vector and the second motion vector may be estimated for each block unit of the frame in the estimator 200.

The generator 300 may select any one of the first motion vector and the second motion vector. The generator 300 may select a motion vector that is more accurately estimated. The selected motion vector may be an accurate motion vector estimated for the motion of the object.

The generator 300 may modify the selected motion vector and generate an intermediate frame between the current frame and the previous frame using the modified motion vector. In this case, the generator 300 may remove the incorrectly estimated motion vectors that exist after the filtering by detecting the distortion region of the modified motion vector and re-correcting the motion vector of the detected distortion region.

As described above, the interpolation apparatus 100 may generate an intermediate frame using the remodified motion vector, thereby improving the frame rate of the video and providing a motion compensation frame interpolation technique suitable for both linear and nonlinear motion.

FIG. 5 is a detailed block diagram of the estimator shown in FIG. 4.

Referring to FIG. 5, the estimator 200 may include a first estimator 210 and a second estimator 230.

The first estimator 210 may estimate the motion vector using two frames. For example, the first estimator 210 may estimate the first motion vector Vm using the current frame t frame and the previous frame t-1 frame.

The first estimator 210 estimates the first forward motion vector through forward motion estimation based on the current frame t using the previous frame t-1 frame and the current frame t frame. A first backward motion vector is estimated through backward motion estimation based on a previous frame t-1 using a frame t and a previous frame t-1. The first motion vector Vm may be estimated using one of the one backward motion vectors.

The second estimator 230 may estimate the motion vector using three frames. For example, the second estimator 230 is a frame contiguous to any one of a current frame t frame, a previous frame t-1 frame, and a current frame t frame and a previous frame t-1 frame. The second motion vector Vt can be estimated using. The frame continuous to either one may be a previous frame (t-2 frame) of the previous frame (t-1 frame) or a subsequent frame (t + 1 frame) of the current frame (t frame).

Specifically, the second estimator 230 refers to the previous frame t-1 frame using the current frame t frame, the previous frame t-1 frame, and the previous frame t-2 frame. The second forward motion vector can be estimated through forward motion estimation. The second estimator 230 uses a current frame t frame, a previous frame t-1 frame, and a subsequent frame t + 1 frame to refer to a later frame t + 1 frame. The second backward motion vector may be estimated through the motion estimation. The second estimator 230 may select one of the second forward motion vector and the second reverse motion vector to estimate the second motion vector Vt.

The first motion vector Vm and the second motion vector Vt may be estimated for each block unit of the frame in the first estimator 210 and the second estimator 230.

6 to 8 are diagrams for describing a method in which the first estimator estimates motion using two frames.

6 is a diagram illustrating a forward motion estimation method among motion estimation methods using two frames, FIG. 7 is a diagram illustrating a backward motion estimation method among motion estimation methods using two frames, and FIG. 8 is a frame of two frames. 2 is a diagram illustrating a bidirectional motion estimation method among the motion estimation methods using the DMA.

The first estimator 210 may estimate the first forward motion vector through a forward motion estimation method. The first estimator 210 may calculate a first forward motion vector as shown in Equation 4.

는 제1 정방향 움직임 벡터,

는 이전 프레임,

는 현재 프레임을 의미한다. 또한,

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the first forward motion vector,

Is the previous frame,

Means the current frame. Also,

Are motion vector candidates, B and S are blocks and search regions, respectively.

The first estimator 210 may estimate the first backward motion vector through backward motion estimation. The first estimator 210 may calculate a first backward motion vector as shown in Equation 5.

는 제1 역방향 움직임 벡터,

는 이전프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the first reverse motion vector,

Is the previous frame,

Means the current frame. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

The first estimator 210 estimates the first forward motion vector and the first backward motion vector, respectively, and then selects a motion vector that minimizes a sum of absolute difference (SAD) value between the current frame and the previous frame among the two vectors. More accurately the estimated motion vector can be determined.

The first estimator 210 may determine a motion vector having a smaller SAD value among the first forward motion vector and the first backward motion vector. The first estimator 210 may calculate the first motion vector as shown in Equation 7 by using Equation 6.

은 제1 움직임 벡터,

는 이전프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the first motion vector,

Is the previous frame,

Means the current frame. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

)과 이전 프레임(

) 간의 중간 프레임(

)을 기준으로 하는 양방향 움직임 추정 방법을 통해 제1 움직임 벡터를 추정할 수 있다.The first estimator 210 uses a motion vector having a smaller SAD value among the SAD value of the first forward motion vector and the SAD value of the first backward motion vector, to determine the current frame (

) And previous frame (

Middle frame between

) May be used to estimate the first motion vector.

은 제1 움직임 벡터를 구하기 위하여 양방향 움직임 추정 방법의 초기 움직임 벡터로 쓰일 수 있다. 초기 움직임 벡터가 가리키는 부근의 좁은 탐색 영역에서 양방향 움직임 추정 방법을 통해 제1 움직임 벡터

가 결정될 수 있다.A motion vector having a smaller SAD value selected by the first estimator 210 from the first forward motion vector and the first backward motion vector

May be used as an initial motion vector of a bidirectional motion estimation method to obtain a first motion vector. First motion vector through a bidirectional motion estimation method in a narrow navigation region indicated by the initial motion vector.

Can be determined.

The first estimator 210 may calculate a first motion vector as shown in Equation 8.

는 제1 움직임 벡터,

과

는 각각 이전 프레임과 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.here

Is the first motion vector,

and

Denotes the previous frame and the current frame, respectively. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

FIG. 9 is a diagram for describing a method in which the second estimator estimates a motion vector using three frames.

The second estimator 230 may estimate the second forward motion vector through the forward motion estimation method. The second estimator 230 may calculate a second forward motion vector as shown in Equation (9).

는 제2 정방향 움직임 벡터,

는 이전 프레임의 이전 프레임,

는 이전 프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the second forward motion vector,

Is the previous frame of the previous frame,

Is the previous frame,

Means the current frame. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

The second estimator 230 may estimate the second reverse motion vector through a backward motion estimation method. The second estimator 230 may calculate a second reverse motion vector as shown in Equation 10.

는 제2 역방향 움직임 벡터,

는 이전 프레임,

는 현재 프레임,

는 현재 프레임의 이후 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here,

Is the second reverse motion vector,

Is the previous frame,

Is the current frame,

Means a frame after the current frame. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

The second estimator 230 may calculate a sum of absolute difference (SAD) value of the second forward motion vector and the second reverse motion vector.

The second estimator 230 may determine a motion vector having a smaller SAD value among the second forward motion vector and the second reverse motion vector. The second estimator 230 may calculate the second motion vector as shown in Equation 12 using Equation 11.

는 제2 움직임 벡터,

는 이전 프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here,

Is the second motion vector,

Is the previous frame,

Means the current frame. Also

Are motion vector candidates, B and S are blocks and search regions, respectively.

), 이전 프레임(

) 및 이전 프레임의 이전 프레임(

)을 이용하는 경우에는 현재 프레임(

이 기준 프레임이 된다.In the case of a motion estimation method using three frames, the middle frame among the three frames may serve as a reference point for vector estimation. For example, the current frame (

), Previous frame (

) And the previous frame of the previous frame (

), The current frame (

This reference frame is used.

In the motion estimation method using three frames, the center frame among the three frames has pixel information and serves as a reference point, so that a more accurate motion vector can be measured and the performance of the motion estimation method can be improved.

FIG. 10 illustrates an example of a detailed block diagram of the generator illustrated in FIG. 4, FIG. 11 illustrates a distortion region detected by the corrector illustrated in FIG. 10, and FIG. 12 illustrates a distortion elimination process according to re-correction of the corrector. It is a diagram showing.

10 to 11, the generator 300 may include a selector 310, a filter 320, and a modifier 330.

The selector 310 may select one of the first motion vector and the second motion vector for each block unit of the frame, generate a motion vector field (MVF), and output the MVF.

The selector 310 may select the first motion vector when the movement of the object in the frame is nonlinear, and select the second motion vector when the movement of the object in the frame is linear. The movement of an object in a frame can be divided into linear or nonlinear based on the movement of the object in the frame during three frames.

The selector 310 may select a motion vector more accurately estimated from the first motion vector and the second motion vector. The motion vector estimated more accurately is a motion vector estimated by a motion estimation method using three frames, and the middle frame serves as a reference point with pixel information. For this reason, the motion estimation method using three frames shows good performance for motion vector estimation when an object moves linearly over three frames. In the motion estimation method using three frames, when the object moves nonlinearly during three frames, the estimation performance of the motion vector is deteriorated. If the object is moving non-linearly, the motion estimation method using two frames shows better performance of motion vector estimation. Therefore, when the motion of the object is linear for several frames, the second motion vector estimated by the motion estimation method using three frames is selected. In addition, when the motion of the object is non-linear, the first motion vector estimated by the motion estimation method using two frames is selected. The MVF, which consists of motion vector blocks selected differently according to the motion pattern of the object, shows more accurate motion estimation performance than the conventional motion estimation method. In this case, since the motion vector selected by the selector 310 selects the motion vector according to the motion pattern of the object, the motion vector selected by the estimator 200 may be more accurately estimated from the first motion vector and the second motion vector estimated by the estimator 200.

The selector 310 uses the relationship with the already estimated neighboring motion vectors existing around the motion vector estimated by the estimator 200 to select the more accurately estimated motion vector. In the case of the motion vector estimated by the estimator 200 more accurately, the vector value difference from neighboring motion vectors is smaller. Using this, the estimator 200 may select a motion vector estimated more accurately among the estimated first and second motion vectors.

The selector 310 may select one of the first motion vector and the second motion vector estimated by the estimator 200 through the following equation (13).

는 선택기(310)가 선택하는 움직임 벡터,

은 이미 추정된 이웃 블록의 움직임 벡터들을 의미한다.From here

Is the motion vector selected by the selector 310,

Denotes motion vectors of neighboring blocks that have already been estimated.

The selector 310 may configure MVF using the selected motion vectors. The selector 310 may output the MVF to the filter 320.

The filter 320 may output the modified MVF by removing the error of the MVF output from the selector 310 through filtering.

There may be motion vectors incorrectly estimated by the estimator 200 in the MVF output from the selector 310. To correct this, most MCFI methods apply mean filtering and median filtering to motion vectors. When vector filtering is applied, a smoothed motion vector is obtained from the neighboring motion vectors, thereby obtaining a coherent motion vector field (MVF) with reduced error. When filtering using the surrounding motion vector, the SAD value is used as a weight to effectively modify the motion vector.

The filter 320 may obtain a modified motion vector through filtering. The filter 320 may calculate a modified motion vector as shown in 15 using Equation 14 below.

은 수정된 움직임 벡터이고,

는 이웃 8개 블록의 움직임 벡터들을 의미한다.here,

Is the modified motion vector,

Denotes motion vectors of neighboring eight blocks.

The filter 320 may apply the modified motion vector to the MVF to generate the modified MVF.

The modifier 330 may generate an intermediate frame between the current frame and the previous frame using the modified MVF and the intermediate frame.

The modifier 330 detects a distortion region of the intermediate frame based on the modified MVF, generates a modified MVF by modifying a motion vector of the modified MVF corresponding to the distortion region, and uses the current frame by using the modified MVF. And an intermediate frame between the previous frame can be created. When the modifier 330 detects a distortion region of an intermediate frame based on the modified MVF, the modified distortion region may be detected using the modified MVF and the intermediate frame based on the modified MVF.

The motion vector correction in the modifier 330 may remove distortions in which incorrectly estimated motion vectors remaining after the filtering in the existing MCFI process occur on an intermediate frame. The modifier 330 first detects in which region of the intermediate frame the distortion occurred based on the modified MVF. Then, by modifying only the motion vectors corresponding to the region where the modifier 330 detects the distortion, it is possible to effectively remove the incorrectly estimated motion vectors remaining after the filtering of the filter 320. In addition, over-smoothing of correctly estimated vectors may not occur due to the recorrection of the modifier 330.

k 사이즈의 n개의 필터를 갖는 convolutional layer를 뜻한다. s는 필터가 이동하는 간격, stride를 의미한다. F(m)은 m개의 뉴런을 갖는 fully connected layer를 뜻한다.The modifier 330 may detect a distortion region through a convolutional neural network (CNN). For example, the CNN structure used by the modifier 330 is C (64,5,1) -C (64,5,1) -C (64,5,1) -C (64,3,1) -F (64) -F (64) -F (2), may be the same. In the CNN structure, C (n, k, s) is k

It means a convolutional layer with n filters of k size. s means stride, the interval at which the filter moves. F (m) means a fully connected layer having m neurons.

The learned network detects in which region distortion occurs as shown in FIG. 11 when inputting the modified MVF and the intermediate frame based on the modified MVF. The distortion area is an area where the points of FIG. 11 exist.

The motion vector corresponding to the region determined by the modifier 330 as distortion through the CNN is corrected using only the peripheral motion vectors determined by the modifier 330 as not a distortion region.

The modifier 330 may obtain the re-corrected motion vector by re-correcting the motion vector modified by the filter 320. The recorrected motion vector can be obtained as shown in Equation 17 using Equation 16.

는 재수정된 움직임 벡터,

는 왜곡이 아닌 것으로 판별된 이웃 블록의 움직임 벡터들을 의미한다.

The redefined motion vector,

Denotes motion vectors of the neighboring block determined as not distortion.

The re-modification of the modifier 330 may have one to five repetitions, but is not limited thereto, and the number of re-modification of the modifier 330 may be n or more (n is a natural number of 1 or more). Through the re-correction process, it is possible to effectively remove the distortion region without the problem of over smoothing the correctly estimated vector as shown in FIG. 12. Numbers shown in Figure 12 refers to the number of re-modification iteration of the modifier 330.

13 is a flowchart illustrating an interpolation method, according to an exemplary embodiment.

Referring to FIG. 13, the estimator 200 estimates a first motion vector using a current frame and a previous frame (1610a). The first motion vector may be estimated for each block unit of a frame.

The estimator 200 estimates the second motion vector by further using a frame consecutive to either the current frame or the previous frame (1610b). The second motion vector may be estimated for each block unit of the frame.

The generator 300 generates an intermediate frame between the current frame and the previous frame by using any one of the first motion vector and the second motion vector (1620).

FIG. 14 is a diagram illustrating average PSNR and SSIM measurement results for each sequence, FIG. 15 is a diagram illustrating a result of a mobile sequence, FIG. 16 is a diagram illustrating a soccer sequence, and FIG. 17 is a result of a Stefan sequence. Drawing.

Experiments were performed using benchmark video sequences to evaluate the performance of the interpolation method according to the embodiment. The sequences used in the experiments were News, Stefan, City, Coastguard, Foreman, Mother & daughter, Mobile, and Soccer in Common intermediate format (CIF). Even frames of these sequences are removed and regenerated by MCFI methods. The performance of the MCFI method can be measured by comparing the generated intermediate frame with the original even-numbered frame.

40 이다. 움직임 수정 방법에 사용된 CNN은 Caffe package를 통해 구현되었다. 일 실시예에 따른 보간 방법은 기존의 다른 보간 방법들, dual ME, novel TME, MBOH, Kim’s, Lim’s과 비교하였다.In the interpolation method according to the embodiment, the block size is 8x8, and the search area is

40 is. The CNN used for motion correction is implemented through the Caffe package. The interpolation method according to an embodiment is compared with other existing interpolation methods, dual ME, novel TME, MBOH, Kim's, and Lim's.

For objective evaluation, average peak signal to noise ratio (PSNR) and structural similarity (SSIM) were measured for each video sequence, and the results are shown in FIG. 14. Through this, it can be confirmed that the performance of the interpolation method according to the embodiment is better than other MCFI methods.

Intermediate frames generated by each MCFI method are compared for subjective evaluation, and the results are shown in FIGS. 15 to 17. It can be seen that the intermediate frames generated through the interpolation method according to the embodiment most closely resemble the original frames. 15 to 17 are residual error maps between an intermediate frame, an intermediate frame, and an original frame generated through each MCFI method. 15 to 17 (a) is an original frame, (b) is a novel TME, (c) is MBOH, (d) is Kim's method, (e) is Lim's method, and (f) is proposed method.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and may configure the processing device to operate as desired, or process independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be embodied permanently or temporarily in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

Although the embodiments have been described with reference to the accompanying drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even by substitution or replacement by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the following claims.

Claims (20)

An estimator for estimating a first motion vector using a current frame and a previous frame and further estimating a second motion vector using a frame continuous to any one of the current frame and the previous frame; And
A generator for generating an intermediate frame between the current frame and the previous frame by using any one of the first motion vector and the second motion vector.
Including,
And the first motion vector and the second motion vector are estimated for each block unit of a frame.
The method of claim 1,
The estimator,
Estimating a first forward motion vector using forward motion estimation based on the current frame using the previous frame and the current frame,
Estimating a first backward motion vector by using backward motion estimation based on the previous frame using the current frame and the previous frame,
Estimating the first motion vector using any one of the first forward motion vector and the first backward motion vector
Interpolation device.
The method of claim 2,
The estimator,
Calculating a sum of absolute difference (SAD) value of the first forward motion vector and the first backward motion vector,
Estimating the first motion vector using a motion vector having a smaller SAD value from a sum of absolute difference (SAD) value of the first forward motion vector and a SAD value of the first backward motion vector;
Interpolation device.
The method of claim 3,
The estimator,
A bidirectional motion estimation method based on an intermediate frame between the current frame and the previous frame using a motion vector having a smaller SAD value among the SAD value of the first forward motion vector and the SAD value of the first backward motion vector. To estimate the first motion vector through
Interpolation device.
The method of claim 1,
The estimator,
Estimating a second forward motion vector by using forward motion estimation based on the previous frame using the current frame, the previous frame, and the previous frame of the previous frame,
Estimating a second backward motion vector by using backward motion estimation based on the next frame using the current frame, the previous frame, and a subsequent frame of the current frame,
Estimating the second motion vector by selecting any one of the second forward motion vector and the second reverse motion vector;
Interpolation device.
The method of claim 5,
The estimator,
Calculating a sum of absolute difference (SAD) value of the second forward motion vector and the second backward motion vector,
Estimating the second motion vector using a motion vector having a smaller SAD value from a sum of absolute difference (SAD) value of the second forward motion vector and a SAD value of the second backward motion vector;
Interpolation device.
The method of claim 1,
The generator,
A selector for selecting one of the first motion vector and the second motion vector for each block unit of a frame and outputting a motion vector field (MVF);
A filter for outputting a modified MVF by removing an error from the MVF; And
A modifier that generates an intermediate frame based on the modified MVF and an intermediate frame between the current frame and the previous frame using the modified MVF
Interpolation device comprising a.
The method of claim 7, wherein
The selector,
If the motion of the object in the frame is nonlinear, select the first motion vector,
When the motion of the object in the frame is linear, selecting the second motion vector
Interpolation device.
The method of claim 7, wherein
The modifier,
Detects a distortion region of an intermediate frame based on the modified MVF, modifies a motion vector of the modified MVF corresponding to the distortion region, generates a modified MVF, and uses the current MVF and the modified MVF To generate an intermediate frame between previous frames
Interpolation device
The method of claim 7, wherein
The modifier,
Detecting Distorted Areas Through Convolutional Neural Networks (CNNs)
Interpolation device.
Estimating a first motion vector using the current frame and the previous frame;
Estimating a second motion vector further using a frame consecutive to any one of the current frame and the previous frame; And
Generating an intermediate frame between the current frame and the previous frame by using any one of the first and second motion vectors.
Including,
And the first motion vector and the second motion vector are estimated for each block unit of a frame.
The method of claim 11,
Estimating the first motion vector,
Estimating a first forward motion vector by using forward motion estimation based on the current frame using the previous frame and the current frame;
Estimating a first backward motion vector using backward motion estimation based on the previous frame using the current frame and the previous frame; And
Estimating the first motion vector using any one of the first forward motion vector and the first backward motion vector.
Interpolation method comprising a.
The method of claim 12,
Estimating the first motion vector using the any one motion vector,
Calculating a sum of absolute difference (SAD) value of the first forward motion vector and the first backward motion vector; And
Estimating the first motion vector using a motion vector having a smaller SAD value from a sum of absolute difference (SAD) value of the first forward motion vector and a SAD value of the first backward motion vector;
Interpolation method comprising a.
The method of claim 13,
Estimating the first motion vector using the motion vector having the smaller SAD value,
A bidirectional motion estimation method based on an intermediate frame between the current frame and the previous frame using a motion vector having a smaller SAD value among the SAD value of the first forward motion vector and the SAD value of the first backward motion vector. Estimating the first motion vector through
Interpolation method comprising a.
The method of claim 11,
Estimating the second motion vector,
Estimating a second forward motion vector through forward motion estimation based on the previous frame using the current frame, the previous frame, and the previous frame of the previous frame;
Estimating a second reverse motion vector by using backward motion estimation based on the next frame using the current frame, the previous frame, and a subsequent frame of the current frame; And
Estimating the second motion vector by selecting one of the second forward motion vector and the second reverse motion vector;
Interpolation method comprising a.
The method of claim 15,
Estimating the second motion vector by selecting the one motion vector;
Calculating a sum of absolute difference (SAD) value of the second forward motion vector and the second backward motion vector; And
Estimating the second motion vector using a motion vector having a smaller SAD value from a sum of absolute difference (SAD) value of the second forward motion vector and a SAD value of the second backward motion vector;
Interpolation method comprising a.
The method of claim 11,
The generating step,
Selecting one of the first motion vector and the second motion vector for each block unit of a frame and outputting a motion vector field (MVF);
A filter removing the error from the MVF to output a modified MVF; And
Generating an intermediate frame based on the modified MVF and the intermediate frame between the current frame and the previous frame using the modified MVF;
Interpolation method comprising a.
The method of claim 17,
The outputting step,
If the motion of the object in the frame is nonlinear, selecting the first motion vector;
If the motion of the object in the frame is nonlinear, selecting the second motion vector
Interpolation method comprising a.
The method of claim 17,
The generating step,
Generating an intermediate frame based on the modified MVF;
Detecting a distortion region of an intermediate frame based on the modified MVF;
Generating a modified MVF by modifying a motion vector of the modified MVF corresponding to the distortion area; And
Generating an intermediate frame between the current frame and the previous frame using the remodified MVF
Interpolation method comprising a.
The method of claim 17,
The detecting step,
Detecting a distortion region of an intermediate frame based on the modified MVF using a convolutional neural network (CNN)
Interpolation method comprising a.

Images