KR102105766B1 - Apparatus and method for motion compensated frame interpolation suitable for both linear and nolinear motion - Google Patents

Abstract

Disclosed is a motion compensation frame interpolation device and method suitable for both linear and nonlinear motion. An interpolation apparatus according to an embodiment estimates a first motion vector using a current frame and a previous frame, and estimates a second motion vector using a frame consecutive to any one of the current frame and the previous frame And a generator that generates an intermediate frame between the current frame and the previous frame using any one of the first motion vector and the second motion vector, and includes the first motion vector and the second motion vector. The vector is estimated for each block unit of the frame.

Description

Apparatus and method for motion compensation frame interpolation suitable for both linear and nonlinear motions {APPARATUS AND METHOD FOR MOTION COMPENSATED FRAME INTERPOLATION SUITABLE FOR BOTH LINEAR AND NOLINEAR MOTION}

The embodiments below relate to a motion compensation frame interpolation apparatus and method suitable for both linear and nonlinear motion.

When there is a bandwidth limitation in the communication system, the frame rate of the video is lowered and transmitted to meet the limitation. In the case of a liquid crystal display (LCD), temporal distortion such as motion blur occurs when the frame rate of the video is low. These problems can be overcome by improving the frame rate of the video in the decoder.

Motion Compensated Frame Interpolation (MCFI) is widely used in decoders to improve the frame rate of video. MCFI is a technique for estimating the motion of an object on a continuous video frame and reflecting it to generate and insert an intermediate frame to improve the frame rate.

In order to improve the frame rate, it is important to accurately estimate the motion of an object on a video frame.

Embodiments provide a technique for accurately estimating the motion of an object on a video frame by selecting an estimated motion vector by a motion estimation method that is more suitable for each motion when the object exhibits linear or nonlinear motion within the frame. You can.

In addition, embodiments can improve the frame rate of a video by providing an intermediate frame using an accurately estimated motion vector, and provide a motion compensation frame interpolation technique suitable for both linear and nonlinear motion.

The interpolation apparatus according to an embodiment estimates a first motion vector using a current frame and a previous frame, and estimates a second motion vector using a frame consecutive to any one of the current frame and the previous frame. And a generator generating an intermediate frame between the current frame and the previous frame by using an estimator and one of the first motion vector and the second motion vector, and the first motion vector and the second motion vector Is estimated for each block unit of the frame.

The estimator estimates a first forward motion vector through forward motion estimation based on the current frame using the previous frame and the current frame, and references the previous frame using the current frame and the previous frame. A first backward motion vector is estimated through backward motion estimation, and the first motion vector can be estimated using any one of the first forward motion vector and the first backward motion vector.

The estimator calculates a sum of absolute difference (SAD) value of the first forward motion vector and the first backward motion vector, and a sum of absolute difference (SAD) value of the first forward motion vector and the first reverse direction. The first motion vector may be estimated using a motion vector having a smaller SAD value among the SAD values of the motion vectors.

The estimator uses a motion vector having a smaller SAD value among the SAD values of the first forward motion vector and the SAD values of the first backward motion vector, based on the middle frame between the current frame and the previous frame. The first motion vector can be estimated through the motion estimation method.

The estimator estimates a second forward motion vector through forward motion estimation based on the previous frame using the current frame, the previous frame, and the previous frame of the previous frame, and the current frame, the previous frame, and A second backward motion vector is estimated through backward motion estimation based on the subsequent frame using a subsequent frame of the current frame, and a motion vector of the second forward motion vector and the second backward motion vector is selected. The second motion vector can be estimated by selection.

The estimator calculates a sum of absolute difference (SAD) value of the second forward motion vector and the second backward motion vector, and a sum of absolute difference (SAD) value of the second forward motion vector and the second reverse direction. The second motion vector may be estimated using a motion vector having a smaller SAD value among the SAD values of the motion vector.

The generator selects one motion vector among the first motion vector and the second motion vector for each block unit of a frame and outputs a motion vector field (MVF), and is corrected by removing an error from the MVF It may include a filter for outputting an MVF, and a modifier for generating an intermediate frame based on the modified MVF and an intermediate frame between the current frame and the previous frame using the modified MVF.

The selector may select the first motion vector when the motion of the object in the frame is nonlinear, and select the second motion vector when the motion of the object in the frame is linear.

The corrector detects a distortion region of an intermediate frame based on the corrected MVF, corrects a motion vector of the corrected MVF corresponding to the distortion region, generates a corrected MVF, and uses the corrected MVF to An intermediate frame between the current frame and the previous frame can be generated.

The corrector may detect a distortion region through a convolutional neural network (CNN).

The interpolation method according to an embodiment may include estimating a first motion vector using a current frame and a previous frame, and further using a second sequence of motion frames using any one of the current frame and the previous frame. Estimating, and generating an intermediate frame between the current frame and the previous frame using any one of the first motion vector and the second motion vector, wherein the first motion vector and the The second motion vector is estimated for each block unit of the frame.

The estimating of the first motion vector may include estimating a first forward motion vector through forward motion estimation based on the current frame using the previous frame and the current frame, and the current frame and the previous Estimating a first backward motion vector through backward motion estimation based on the previous frame using a frame, and using any one of the first forward motion vector and the first backward motion vector And estimating the first motion vector.

The estimating the first motion vector using any one of the motion vectors includes: calculating a sum of absolute difference (SAD) values of the first forward motion vector and the first backward motion vector; And estimating the first motion vector using a motion vector having a smaller SAD value among the sum of absolute difference (SAD) value of the forward motion vector and the SAD value of the first backward motion vector.

The estimating the first motion vector using the motion vector having the smaller SAD value may include: having a smaller SAD value among the SAD values of the first forward motion vector and the SAD values of the first backward motion vector. And estimating a first motion vector through a bidirectional motion estimation method based on an intermediate frame between the current frame and the previous frame using a vector.

The estimating of the second motion vector may include estimating a second forward motion vector through forward motion estimation based on the previous frame using the current frame, the previous frame, and the previous frame of the previous frame. , Estimating a second backward motion vector through backward motion estimation based on the subsequent frame using the current frame, the previous frame, and the subsequent frame of the current frame, and the second forward motion vector and the second And estimating the second motion vector by selecting any one of the two backward motion vectors.

The estimating the second motion vector by selecting any one of the motion vectors includes calculating a sum of absolute difference (SAD) values of the second forward motion vector and the second backward motion vector; And estimating the second motion vector using a motion vector having a smaller SAD value among the sum of absolute difference (SAD) value of the forward motion vector and the SAD value of the second backward motion vector.

In the generating step, a motion vector field (MVF) is output by selecting any one of the first motion vector and the second motion vector for each block unit of a frame, and a filter generates an error in the MVF. The method may include removing and outputting a modified MVF, and generating an intermediate frame based on the modified MVF and an intermediate frame between the current frame and the previous frame using the modified MVF.

The outputting may include selecting the first motion vector when the motion of the object in the frame is nonlinear, and selecting the second motion vector when the motion of the object in the frame is nonlinear. It can contain.

The generating may include generating an intermediate frame based on the modified MVF, detecting a distortion region of the intermediate frame based on the modified MVF, and a motion vector of the modified MVF corresponding to the distortion region. And generating a re-corrected MVF, and generating an intermediate frame between the current frame and the previous frame using the re-corrected MVF.

The detecting may include detecting a distortion region of an intermediate frame based on the modified MVF using a convolutional neural network (CNN).

1 is a view showing a unidirectional motion estimation method.
2 is a diagram illustrating a bidirectional motion estimation method.
3 is an exemplary diagram of an estimated motion vector field.
4 is a schematic block diagram of an interpolation device according to an embodiment.
5 is a detailed block diagram of the estimator illustrated in FIG. 4.
6 is a diagram illustrating a forward motion estimation method among motion estimation methods using two frames.
7 is a diagram illustrating a backward motion estimation method among motion estimation methods using two frames.
8 is a diagram illustrating a bidirectional motion estimation method among motion estimation methods using two frames.
9 is a diagram showing a motion estimation method using three frames.
10 is a detailed block diagram of the generator illustrated in FIG. 4.
11 is a view showing a distortion region detected by a crystal.
12 is a view showing a distortion removal process according to the iterative correction of the corrector.
13 is a flowchart illustrating an interpolation method according to an embodiment.
14 is a view showing the average PSNR and SSIM measurement results for each sequence.
15 is a diagram showing the results of a Mobile sequence.
16 is a view showing the results of a Soccer sequence.
17 is a diagram showing the results of the Stefan sequence.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments, and the scope of the patent application right is not limited or limited by these embodiments. It should be understood that all modifications, equivalents, or substitutes for the embodiments are included in the scope of rights.

The terms used in the examples are for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described on the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

The terms first or second may be used to describe various components, but the components should not be limited by the terms. The terms are for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the embodiment, the first component may be referred to as the second component, and similarly The second component may also be referred to as the first component.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the embodiments, detailed descriptions thereof will be omitted.

1 is a diagram illustrating a unidirectional motion estimation method, FIG. 2 is a diagram illustrating a bidirectional motion estimation method, and FIG. 3 is an exemplary diagram of an estimated motion vector field.

1 to 3, a motion compensated frame interpolation (MCFI) refers to a technique of generating an interpolated frame between consecutive frames to improve a frame rate of video content. . MCFI can be divided into motion estimation, motion vector correction, and intermediate frame generation.

MCFI uses a motion estimation method to estimate the motion of an object between successive frames. At this time, a block matching algorithm (BMA) is mainly used in consideration of appropriate time complexity and performance. The block matching algorithm is a method of dividing an input frame into blocks and then estimating a motion vector for each block.

The motion estimation method is divided into a uni-directional motion estimation method and a bi-directional motion estimation method according to which frame the reference is set to.

The unidirectional motion estimation method may be as illustrated in FIG. 1. The unidirectional motion estimation method moves based on the current frame when generating an interpolated frame (t-1 / 2) between a previous frame (t-1) and a current frame (t-1). Estimate the vector. The motion vector is estimated using equation (1).

는 단방향 움직임 벡터,

는 이전 프레임

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here

Is a one-way motion vector,

The old frame

Means the current frame. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

The unidirectional motion estimation method can estimate a motion vector with high accuracy, but an overlap region and a hole region inevitably occur when generating an intermediate frame.

The bidirectional motion estimation method may be as illustrated in FIG. 2. When generating an interpolated frame (t-1 / 2) between the previous frame (t-1) and the current frame (current frame, t), the motion vector is estimated based on the intermediate frame. The motion vector is estimated using equation (2).

는 양방향 움직임 벡터,

는 이전프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here

Bidirectional motion vector,

Is the previous frame,

Means the current frame. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

In the bidirectional motion estimation method, since motion vectors are obtained based on the intermediate frame (t-1 / 2), an overlap region and a hole region do not occur. In the general bidirectional motion estimation method, the block of the intermediate frame (t-1 / 2) serves as a reference point when estimating motion between the previous frame (t-1) and the current frame (t). However, since the block of the middle frame (t-1 / 2) does not yet have pixel information, it serves as a positional reference point and does not help to find an accurate motion vector. Due to this, the bi-directional motion estimation method often estimates the wrong motion vector and the accuracy of motion vector estimation is deteriorated.

Through the motion estimation process using the unidirectional motion estimation method and the bidirectional motion estimation method, motion vectors are estimated for all blocks of the frame. Block-based motion vectors obtained by the motion estimation method constitute a motion vector field (MVF) as shown in FIG. 3.

Accurate motion vectors are not always estimated in the motion estimation process. Therefore, there are motion vectors that are incorrectly estimated on the MVF and must be corrected. Most MCFI methods apply mean filtering and median filtering to the estimated motion vector. When filtering the estimated motion vector, the motion vector is smoothed through the surrounding motion vectors, and thus a coherent MVF with reduced error can be obtained.

However, some motion vectors are not properly corrected even when filtering is applied. Incorrectly estimated motion vectors that exist even after filtering generate distortion in the region when an intermediate frame is generated. Incorrectly estimated motion vectors that exist even after filtering are not well removed by additional filtering. The incorrectly estimated motion vectors that exist even after filtering excessively over-smoothing the accurately estimated motion vectors when additional filtering is performed.

In the last stage of the MCFI, the intermediate frame generation stage, the intermediate frame is generated through the modified vectors through filtering. At this time, the intermediate frame is generated using pixel values of the previous frame and the current frame corresponding to the position indicated by the corrected motion vector. The intermediate frame is generated using equation (3).

는 중간프레임,

과

는 각각 이전 프레임과 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들을 의미한다.이처럼, MCFI는 비디오 프레임 상의 물체의 움직임을 추정하고 이를 반영하여 중간 프레임을 생성, 삽입함으로써 프레임률을 향상시킨다. 따라서, 프레임 상의 물체의 움직임을 정확히 추정하는 것이 중요하다.here

Is the middle frame,

and

Indicates the previous frame and the current frame, respectively. In addition

Denotes motion vector candidates. In this way, MCFI estimates the motion of an object on a video frame and reflects it to generate and insert an intermediate frame to improve the frame rate. Therefore, it is important to accurately estimate the motion of the object on the frame.

As described above, in the unidirectional motion estimation method, when an intermediate frame is generated, an overlap region and a hole region inevitably occur. In addition, since the bidirectional motion estimation method is to find a motion vector based on an intermediate frame without pixel information, an incorrectly estimated motion vector exists, and thus, the accuracy of motion vector estimation is deteriorated. In the filtering process, incorrectly estimated motion vectors that exist even after filtering excessively smooth the estimated motion vectors.

Embodiments can estimate a more accurate motion vector and use the estimated motion vector to generate an intermediate frame of improved quality. Hereinafter, this will be described in detail with reference to FIGS. 4 to 17.

4 is a schematic block diagram of an interpolation device according to an embodiment.

Referring to FIG. 4, the interpolation device 100 includes an estimator 200 and a generator 300.

The estimator 200 may estimate a motion vector using two or more motion estimation methods using one or more frames. For the motion estimation method, a motion estimation method using two frames and a motion estimation method using three frames can be used.

The estimator 200 may estimate the first motion vector using two frames, and estimate the second motion vector using three frames. The first motion vector and the second motion vector may be estimated for each block unit of the frame in the estimator 200.

The generator 300 may select any one motion vector from the first motion vector and the second motion vector. The generator 300 may more accurately select the estimated motion vector. The selected motion vector may be a highly accurate motion vector estimated for the motion of the object.

The generator 300 may modify the selected motion vector and use the modified motion vector to generate an intermediate frame between the current frame and the previous frame. At this time, the generator 300 detects a distortion region of the corrected motion vector, and re-corrects the detected motion vector of the distortion region, thereby removing erroneously estimated motion vectors that exist even after filtering.

As described above, the interpolation apparatus 100 may improve the frame rate of the video and provide a motion compensation frame interpolation technique suitable for both linear and nonlinear motion by generating an intermediate frame using the re-corrected motion vector.

5 is a detailed block diagram of the estimator illustrated in FIG. 4.

Referring to FIG. 5, the estimator 200 may include a first estimator 210 and a second estimator 230.

The first estimator 210 may estimate a motion vector using two frames. For example, the first estimator 210 may estimate the first motion vector Vm using the current frame (t frame) and the previous frame (t-1 frame).

The first estimator 210 estimates the first forward motion vector through forward motion estimation based on the current frame (t frame) using the previous frame (t-1 frame) and the current frame (t frame). A first backward motion vector is estimated through backward motion estimation based on a previous frame (t-1 frame) using a frame (t frame) and a previous frame (t-1 frame). The first motion vector Vm may be estimated using any one of the one backward motion vectors.

The second estimator 230 may estimate the motion vector using three frames. For example, the second estimator 230 is a frame that is continuous to any one of the current frame (t frame), the previous frame (t-1 frame), and the current frame (t frame) and the previous frame (t-1 frame). The second motion vector Vt can be estimated using. The frame continuous to either one may be a previous frame (t-2 frame) of the previous frame (t-1 frame) or a subsequent frame (t + 1 frame) of the current frame (t frame).

Specifically, the second estimator 230 references the previous frame (t-1 frame) using the current frame (t frame), the previous frame (t-1 frame), and the previous frame (t-2 frame) of the previous frame. The second forward motion vector can be estimated through the forward motion estimation as. The second estimator 230 uses a current frame (t frame), a previous frame (t-1 frame), and a subsequent frame (t + 1 frame) of the current frame, based on a subsequent frame (t + 1 frame). The second backward motion vector may be estimated through motion estimation. The second estimator 230 may estimate the second motion vector Vt by selecting any one of the second forward motion vector and the second backward motion vector.

The first motion vector Vm and the second motion vector Vt may be estimated for each block unit of a frame in the first estimator 210 and the second estimator 230.

6 to 8 are diagrams for explaining a method in which the first estimator estimates motion using two frames.

FIG. 6 is a diagram showing a forward motion estimation method among motion estimation methods using two frames, FIG. 7 is a view showing a reverse motion estimation method among motion estimation methods using two frames, and FIG. 8 is two frames It is a view showing a bidirectional motion estimation method among motion estimation methods using.

The first estimator 210 may estimate the first forward motion vector through the forward motion estimation method. The first estimator 210 may calculate the first forward motion vector as shown in Equation (4).

는 제1 정방향 움직임 벡터,

는 이전 프레임,

는 현재 프레임을 의미한다. 또한,

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the first forward motion vector,

The old frame,

Means the current frame. In addition,

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

The first estimator 210 may estimate the first backward motion vector through backward motion estimation. The first estimator 210 may calculate a first backward motion vector as shown in Equation (5).

는 제1 역방향 움직임 벡터,

는 이전프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the first reverse motion vector,

Is the previous frame,

Means the current frame. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

The first estimator 210 estimates the first forward motion vector and the first backward motion vector, respectively, and then selects a motion vector that minimizes a sum of absolute difference (SAD) value between the current frame and the previous frame from the two vectors. A more accurately estimated motion vector can be determined.

The first estimator 210 may determine a motion vector having a smaller SAD value among the first forward motion vector and the first backward motion vector. The first estimator 210 may calculate the first motion vector as in Equation 7 using Equation 6.

은 제1 움직임 벡터,

는 이전프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the first motion vector,

Is the previous frame,

Means the current frame. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

)과 이전 프레임(

) 간의 중간 프레임(

)을 기준으로 하는 양방향 움직임 추정 방법을 통해 제1 움직임 벡터를 추정할 수 있다.The first estimator 210 uses the motion vector having the smaller SAD value among the SAD value of the first forward motion vector and the SAD value of the first backward motion vector (

) And previous frame (

) Between intermediate frames (

The first motion vector may be estimated through a bidirectional motion estimation method based on).

은 제1 움직임 벡터를 구하기 위하여 양방향 움직임 추정 방법의 초기 움직임 벡터로 쓰일 수 있다. 초기 움직임 벡터가 가리키는 부근의 좁은 탐색 영역에서 양방향 움직임 추정 방법을 통해 제1 움직임 벡터

가 결정될 수 있다.The motion vector having a smaller SAD value selected by the first estimator 210 from the first forward motion vector and the first reverse motion vector

Can be used as an initial motion vector of a bidirectional motion estimation method to obtain a first motion vector. The first motion vector through a bidirectional motion estimation method in a narrow search area near the initial motion vector

Can be determined.

The first estimator 210 may calculate the first motion vector as shown in Equation (8).

는 제1 움직임 벡터,

과

는 각각 이전 프레임과 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.here

Is the first motion vector,

and

Indicates the previous frame and the current frame, respectively. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

FIG. 9 is a diagram for explaining a method of estimating a motion vector by using a three frame estimator.

The second estimator 230 may estimate the second forward motion vector through the forward motion estimation method. The second estimator 230 may calculate a second forward motion vector as shown in Equation (9).

는 제2 정방향 움직임 벡터,

는 이전 프레임의 이전 프레임,

는 이전 프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.

Is the second forward motion vector,

Is the previous frame of the previous frame,

The old frame,

Means the current frame. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

The second estimator 230 may estimate the second backward motion vector through the backward motion estimation method. The second estimator 230 may calculate a second backward motion vector as shown in Equation (10).

는 제2 역방향 움직임 벡터,

는 이전 프레임,

는 현재 프레임,

는 현재 프레임의 이후 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here,

Is the second reverse motion vector,

The old frame,

Is the current frame,

Means a frame after the current frame. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

The second estimator 230 may calculate sum of absolute difference (SAD) values of the second forward motion vector and the second backward motion vector.

The second estimator 230 may determine a motion vector having a smaller SAD value among the second forward motion vector and the second backward motion vector. The second estimator 230 may calculate the second motion vector as shown in Equation 12 using Equation (11).

는 제2 움직임 벡터,

는 이전 프레임,

는 현재 프레임을 의미한다. 또한

는 움직임 벡터 후보들, B와 S는 각각 블록과 탐색 영역을 의미한다.From here,

Is the second motion vector,

The old frame,

Means the current frame. In addition

Denotes motion vector candidates, and B and S denote blocks and search regions, respectively.

), 이전 프레임(

) 및 이전 프레임의 이전 프레임(

)을 이용하는 경우에는 현재 프레임(

이 기준 프레임이 된다.In the case of a motion estimation method using three frames, a middle frame among the three frames may serve as a reference point for vector estimation. For example, the current frame (

), Previous frame (

) And the previous frame of the previous frame (

), The current frame (

This is the reference frame.

In the motion estimation method using three frames, the middle frame of the three frames has pixel information and serves as a reference point, so more accurate motion vectors can be measured and contribute to improving the performance of the motion estimation method.

FIG. 10 shows an example of a specific block diagram of the generator shown in FIG. 4, FIG. 11 is a view showing a distortion area detected by the crystal shown in FIG. 10, and FIG. 12 is a process for removing distortion according to re-correction of the corrector It is a view showing.

10 to 11, the generator 300 may include a selector 310, a filter 320, and a modifier 330.

The selector 310 may select one of the first motion vector and the second motion vector for each block unit of the frame to generate a motion vector field (MVF), and output the MVF.

The selector 310 may select a first motion vector when the motion of the object in the frame is nonlinear, and select a second motion vector when the motion of the object in the frame is linear. The movement of an object in a frame can be divided into linear or nonlinear based on the movement of an object in a frame during three frames.

The selector 310 may select a more accurately estimated motion vector from the first motion vector and the second motion vector. The more accurately estimated motion vector is a motion vector estimated by a motion estimation method using three frames, and a middle frame serves as a reference point with pixel information. For this reason, the motion estimation method using three frames shows good performance for motion vector estimation when the object moves linearly during three frames. In the motion estimation method using three frames, when the object moves nonlinearly during three frames, the estimation performance of the motion vector is deteriorated. When the object is moving nonlinearly, the motion estimation method using two frames shows better motion vector estimation performance. Therefore, when the motion of the object is linear during several frames, a second motion vector estimated by a motion estimation method using three frames is selected. In addition, when the motion of the object is nonlinear, the first motion vector estimated by the motion estimation method using two frames is selected. MVF composed of motion vector blocks selected differently according to the motion pattern of the object has more accurate motion estimation performance than the existing motion estimation method. At this time, since the motion vector selected by the selector 310 selects a motion vector according to the motion pattern of the object, it may be said that the motion vector is more accurately estimated from the first motion vector and the second motion vector estimated by the estimator 200.

The selector 310 uses the relationship with the already estimated neighboring motion vectors existing around the motion vector estimated by the estimator 200 in order to more accurately select the estimated motion vector. In the case of a motion vector estimated by the estimator 200 more accurately, a difference in vector values from neighboring motion vectors is less. If this is used, a more accurately estimated motion vector can be selected from the first and second motion vectors estimated by the estimator 200.

The selector 310 may select one of the first motion vector and the second motion vector estimated by the estimator 200 through Equation 13 below.

는 선택기(310)가 선택하는 움직임 벡터,

은 이미 추정된 이웃 블록의 움직임 벡터들을 의미한다.From here

Is a motion vector selected by the selector 310,

Denotes motion vectors of neighboring blocks that have already been estimated.

The selector 310 may be configured as an MVF using selected motion vectors. The selector 310 may output the MVF to the filter 320.

The filter 320 may output the corrected MVF by removing the error of the MVF output from the selector 310 through filtering.

The MVFs output from the selector 310 may include motion vectors that the estimator 200 incorrectly estimates. To correct this, most MCFI methods apply mean filtering and median filtering to motion vectors. When vector filtering is applied in this way, a smoothed motion vector is obtained through neighboring motion vectors, and thus a coherent motion vector field (MVF) with reduced error can be obtained. When filtering is performed using a surrounding motion vector, effective motion vector correction can be performed by using the SAD value as a weight.

The filter 320 may obtain a corrected motion vector through filtering. The filter 320 may calculate the corrected motion vector as in 15 using Equation 14 below.

은 수정된 움직임 벡터이고,

는 이웃 8개 블록의 움직임 벡터들을 의미한다.here,

Is a modified motion vector,

Denotes motion vectors of 8 neighboring blocks.

The filter 320 may generate a modified MVF by applying the modified motion vector to the MVF.

The modifier 330 may generate an intermediate frame between the current frame and the previous frame using the modified MVF and the intermediate frame based on the modified MVF.

The corrector 330 detects the distortion region of the intermediate frame based on the modified MVF, corrects the motion vector of the modified MVF corresponding to the distortion region, generates a re-corrected MVF, and uses the re-corrected MVF to make the current frame And an intermediate frame between the previous frames. When the corrector 330 detects the distortion region of the intermediate frame based on the modified MVF, the corrected distortion region may be detected using the modified MVF and the intermediate frame based on the modified MVF.

The correction of the motion vector in the corrector 330 may remove distortion occurring in the middle frame of the incorrectly estimated motion vectors remaining after filtering during the existing MCFI process. The corrector 330 first detects which region of the intermediate frame based on the modified MVF has caused distortion. Then, by correcting only the motion vectors corresponding to the region in which the corrector 330 detects distortion, it is possible to effectively remove the remaining incorrectly estimated motion vectors even after the filter 320 is filtered. In addition, over-smoothing of accurately estimated vectors may not occur due to the re-correction of the corrector 330.

k 사이즈의 n개의 필터를 갖는 convolutional layer를 뜻한다. s는 필터가 이동하는 간격, stride를 의미한다. F(m)은 m개의 뉴런을 갖는 fully connected layer를 뜻한다.The corrector 330 may detect a distortion region through a convolutional neural network (CNN). For example, the CNN structure used by the modifier 330 is C (64,5,1) -C (64,5,1) -C (64,5,1) -C (64,3,1) -F (64) -F (64) -F (2). In CNN structure, C (n, k, s) is k

It means a convolutional layer with n filters of size k. s means stride, the interval at which the filter moves. F (m) means a fully connected layer with m neurons.

When the learned network inputs the modified MVF and the intermediate frame based on the modified MVF, as shown in FIG. 11, it detects in which region distortion occurs. The distortion area is an area in which the dots in FIG. 11 exist.

The motion vector corresponding to the region determined by the corrector 330 to be distortion through the CNN is corrected using only neighboring motion vectors determined by the corrector 330 to be not the distortion region.

The modifier 330 may re-modify the modified motion vector in the filter 320 to obtain a re-modified motion vector. The re-corrected motion vector can be obtained using Equation (16) as Equation (17).

는 재수정된 움직임 벡터,

는 왜곡이 아닌 것으로 판별된 이웃 블록의 움직임 벡터들을 의미한다.

Is a revised motion vector,

Denotes motion vectors of neighboring blocks determined to be not distortion.

The re-modification of the modifier 330 may have 1 to 5 repetitions, but is not limited thereto, and the re-modification of the modifier 330 may be n or more times (n is a natural number of 1 or more). Through the re-correction process, it is possible to effectively remove the distortion region without over smoothing the vector accurately estimated as shown in FIG. 12. The numbers shown in FIG. 12 refer to the number of re-modification repetitions of the corrector 330.

13 is a flowchart illustrating an interpolation method according to an embodiment.

Referring to FIG. 13, the estimator 200 estimates a first motion vector using a current frame and a previous frame (1610a). The first motion vector may be estimated for each block unit of the frame.

The estimator 200 estimates the second motion vector by further using a frame that is continuous to either the current frame or the previous frame (1610b). The second motion vector may be estimated for each block unit of the frame.

The generator 300 generates an intermediate frame between the current frame and the previous frame using one of the first motion vector and the second motion vector (1620).

14 is a view showing the average PSNR and SSIM measurement results for each sequence, FIG. 15 is a view showing the results of a mobile sequence, FIG. 16 is a view showing the results of a soccer sequence, and FIG. 17 is a result of the Stefan sequence It is a drawing.

In order to evaluate the performance of the interpolation method according to the embodiment, experiments were performed using benchmark video sequences. The sequences used in the experiment are News, Stefan, City, Coastguard, Foreman, Mother & daughter, Mobile, Soccer in Common intermediate format (CIF). Even-numbered frames of these sequences are removed and then regenerated by MCFI methods. The performance of the MCFI method can be measured by comparing the generated intermediate frame with the original even frame.

40 이다. 움직임 수정 방법에 사용된 CNN은 Caffe package를 통해 구현되었다. 일 실시예에 따른 보간 방법은 기존의 다른 보간 방법들, dual ME, novel TME, MBOH, Kim’s, Lim’s과 비교하였다.In the interpolation method according to the embodiment, a block size of 8x8 was used, and the search area was

It is 40. The CNN used for the motion correction method was implemented through the Caffe package. The interpolation method according to one embodiment was compared with other existing interpolation methods, dual ME, novel TME, MBOH, Kim's, and Lim's.

For objective evaluation, average peak signal to noise ratio (PSNR) and structural similarity (SSIM) were measured and compared for each video sequence, and the results are shown in FIG. 14. Through this, it can be confirmed that the performance of the interpolation method according to the embodiment is better compared to other MCFI methods.

For subjective evaluation, intermediate frames generated through each MCFI method were compared, and the results are shown in FIGS. 15 to 17. It can be confirmed that the intermediate frames generated through the interpolation method according to the embodiment are most similar to the original frame. 15 to 17 are residual error maps between intermediate frames, intermediate frames and original frames generated through each MCFI method. 15A to 17A are original frames, (b) novel TME, (c) MBOH, (d) Kim's method, (e) Lim's method, and (f) proposed method.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

An estimator for estimating a first motion vector using a current frame and a previous frame, and estimating a second motion vector using a frame consecutive to any one of the current frame and the previous frame; And
Generator for generating an intermediate frame between the current frame and the previous frame using any one of the first motion vector and the second motion vector
Including,
The first motion vector and the second motion vector are estimated for each block unit of the frame,
The generator is an interpolation device that selects any one of the first motion vector and the second motion vector according to whether the motion of the object in the frame is linear or nonlinear.
According to claim 1,
The estimator,
Estimating a first forward motion vector through forward motion estimation based on the current frame using the previous frame and the current frame,
A first backward motion vector is estimated through backward motion estimation based on the previous frame using the current frame and the previous frame,
Estimating the first motion vector using any one of the first forward motion vector and the first reverse motion vector
Interpolation device.
According to claim 2,
The estimator,
Calculate the sum of absolute difference (SAD) values of the first forward motion vector and the first reverse motion vector,
Estimating the first motion vector using a motion vector having a smaller SAD value among the sum of absolute difference (SAD) value of the first forward motion vector and the SAD value of the first backward motion vector
Interpolation device.
According to claim 3,
The estimator,
A bidirectional motion estimation method based on an intermediate frame between the current frame and the previous frame using a motion vector having a smaller SAD value among the SAD values of the first forward motion vector and the SAD values of the first backward motion vector To estimate the first motion vector through
Interpolation device.
According to claim 1,
The estimator,
Estimating a second forward motion vector through forward motion estimation based on the previous frame using the current frame, the previous frame, and the previous frame of the previous frame,
A second backward motion vector is estimated through backward motion estimation based on the subsequent frame using the current frame, the previous frame, and the subsequent frame of the current frame,
Estimating the second motion vector by selecting any one of the second forward motion vector and the second backward motion vector
Interpolation device.
The method of claim 5,
The estimator,
Calculate the sum of absolute difference (SAD) values of the second forward motion vector and the second backward motion vector,
Estimating the second motion vector using a motion vector having a smaller SAD value among the sum of absolute difference (SAD) value of the second forward motion vector and the SAD value of the second backward motion vector
Interpolation device.
According to claim 1,
The generator,
A selector which selects one motion vector among the first motion vector and the second motion vector for each block unit of a frame and outputs a motion vector field (MVF);
A filter that removes the error from the MVF and outputs a corrected MVF; And
A modifier that generates an intermediate frame based on the modified MVF and an intermediate frame between the current frame and the previous frame using the modified MVF
Interpolation device comprising a.
The method of claim 7,
The selector,
If the motion of the object in the frame is nonlinear, select the first motion vector,
When the motion of the object in the frame is linear, selecting the second motion vector
Interpolation device.
The method of claim 7,
The corrector,
A distortion region of an intermediate frame based on the modified MVF is detected, and a revised MVF is generated by correcting a motion vector of the modified MVF corresponding to the distortion region, and the current frame and the modified MVF are used. To create an intermediate frame between previous frames
Interpolation device
The method of claim 7,
The corrector,
Detecting distortion regions through a convolutional neural network (CNN)
Interpolation device.
Estimating a first motion vector using the current frame and the previous frame;
Estimating a second motion vector by further using a frame consecutive to any one of the current frame and the previous frame; And
Generating an intermediate frame between the current frame and the previous frame using any one of the first motion vector and the second motion vector
Including,
The first motion vector and the second motion vector are estimated for each block unit of the frame,
The generating step,
Selecting one of the first motion vector and the second motion vector according to whether the motion of the object in the frame is linear or nonlinear.
Interpolation method comprising a.
The method of claim 11,
Estimating the first motion vector,
Estimating a first forward motion vector through forward motion estimation based on the current frame using the previous frame and the current frame;
Estimating a first backward motion vector through backward motion estimation based on the previous frame using the current frame and the previous frame; And
Estimating the first motion vector using any one of the first forward motion vector and the first reverse motion vector
Interpolation method comprising a.
The method of claim 12,
Estimating the first motion vector using the one of the motion vectors,
Calculating a sum of absolute difference (SAD) value of the first forward motion vector and the first backward motion vector; And
Estimating the first motion vector using a motion vector having a smaller SAD value among the sum of absolute difference (SAD) value of the first forward motion vector and the SAD value of the first backward motion vector.
Interpolation method comprising a.
The method of claim 13,
Estimating the first motion vector using the motion vector having the smaller SAD value,
A bidirectional motion estimation method based on an intermediate frame between the current frame and the previous frame using a motion vector having a smaller SAD value among the SAD values of the first forward motion vector and the SAD values of the first backward motion vector Estimating the first motion vector through
Interpolation method comprising a.
The method of claim 11,
Estimating the second motion vector,
Estimating a second forward motion vector through forward motion estimation based on the previous frame using the current frame, the previous frame, and the previous frame of the previous frame;
Estimating a second backward motion vector through backward motion estimation based on the subsequent frame using the current frame, the previous frame, and the subsequent frame of the current frame; And
Estimating the second motion vector by selecting any one of the second forward motion vector and the second backward motion vector
Interpolation method comprising a.
The method of claim 15,
The estimating the second motion vector by selecting the one motion vector may include:
Calculating a sum of absolute difference (SAD) value of the second forward motion vector and the second backward motion vector; And
Estimating the second motion vector using a motion vector having a smaller SAD value among the sum of absolute difference (SAD) value of the second forward motion vector and the SAD value of the second backward motion vector.
Interpolation method comprising a.
The method of claim 11,
The generating step,
Outputting a motion vector field (MVF) by selecting one motion vector among the first motion vector and the second motion vector for each block unit of a frame;
A filter removing the error from the MVF and outputting the corrected MVF; And
Generating an intermediate frame based on the modified MVF and an intermediate frame between the current frame and the previous frame using the modified MVF;
Interpolation method comprising a.
The method of claim 17,
The step of outputting,
When the motion of the object in the frame is nonlinear, selecting the first motion vector;
If the motion of the object in the frame is nonlinear, selecting the second motion vector
Interpolation method comprising a.
The method of claim 17,
The generating step,
Generating an intermediate frame based on the modified MVF;
Detecting a distortion region of an intermediate frame based on the modified MVF;
Generating a re-corrected MVF by correcting a motion vector of the modified MVF corresponding to the distortion region; And
Generating an intermediate frame between the current frame and the previous frame using the revised MVF
Interpolation method comprising a.
The method of claim 19,
The detecting step,
Detecting a distortion region of an intermediate frame based on the modified MVF using a convolutional neural network (CNN)
Interpolation method comprising a.

Images