KR101215128B1 - Method for frame interpolation - Google Patents

Abstract

A frame interpolation method is disclosed. First, a plurality of preliminary frames are generated through block position based frame interpolation and motion vector position based frame interpolation using the first and second frames that are temporally contiguous, and the generated plurality of preliminary frames are merged into one merge. After generating the frame, the final interpolation frame is generated by interpolating the occlusion sections included in the generated merged frame. Therefore, the accuracy of frame interpolation can be improved, and the quality of the interpolated image can be improved.

Description

Frame interpolation method {METHOD FOR FRAME INTERPOLATION}

The present invention relates to a frame rate up conversion (FRUC) technique of video, and more particularly, to a frame interpolation method capable of improving image quality.

Recently, with the development of mobile communication technology and video compression technology, communication and broadcasting have been converged, and with the rapid development of display devices, the demand for video formats of various sizes and large amounts of high resolution video has increased.

However, it is practically difficult to transmit high resolution data without loss in a limited bandwidth, and various frame rate up conversion techniques have been used to solve this problem.

Frame rate up-conversion is a technique of increasing the frame rate of an image by using an existing frame. By providing more images for the same time, a screen drag phenomenon can be solved by reducing the time to maintain pixel values per frame.

Representative methods of frame rate up-conversion include adding successive frames and interpolating an image using motion compensation.

The method of adding continuous frames does not take into account the motion of the object, so it is low in complexity and easily interpolated. However, when the format of an image is expanded or the movement of an object is large, ghost artefact and screen drag phenomenon may occur in which the image is uneven and the image is overlapped. In addition, since interpolation is performed on a block basis, a block artefact occurs at a boundary between blocks.

Meanwhile, a method of interpolating an image by using motion compensation is a method of generating a new image by predicting a motion of an object in a neighboring frame. This method requires more computational complexity than adding continuous frames, but it can solve problems such as screen dragging. However, since the image interpolation method using motion compensation performs prediction on the assumption that the selected block moves linearly in the current frame using the motion vector predicted in the previous frame, if the prediction of the motion vector is not accurate, The motion vector may also select a wrong position, which may cause serious blocking in the interpolated frame.

An object of the present invention for solving the above problems is to provide a frame interpolation method that can improve the accuracy of the frame interpolation, and can improve the quality of the image.

Technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

Frame interpolation method according to an aspect of the present invention for achieving the above object of the present invention through the block position-based frame interpolation and motion vector position-based frame interpolation using the first frame and the second frame in time Generating a plurality of preliminary frames, merging the generated plurality of preliminary frames to generate a single merge frame, and generating a final interpolated frame by interpolating the occlusion sections included in the generated merged frames. Include.

The generating of the plurality of preliminary frames may include searching for a reference block corresponding to a current block of the second frame through a global search in the first frame, and based on a peak signal to noise ratio (PSNR). Setting a threshold, comparing a SAD of the reference block found through the global search with the threshold, and a reference block whose SAD of the candidate block is smaller than the threshold and a current block corresponding to the reference block And generating the interpolation block through the block position based frame interpolation and the motion vector based frame interpolation, for generating the plurality of preliminary frames.

The generating of the plurality of preliminary frames may generate two preliminary frames through the bidirectional block position based frame interpolation, and may generate two preliminary frames through the bidirectional motion vector based frame interpolation.

In the generating of the plurality of preliminary frames, the block position based frame interpolation may be set to have a lower threshold value than the motion vector based frame interpolation by applying the weight of the threshold value.

The setting of the threshold value based on the PSNR may include calculating a sum of square difference (SSD) by calculating an error of each block from the PSNR value, and comparing the SSD and sum of absolute difference (SAD). And applying the correction coefficient to the SSD to set the threshold value.

The generating of the merge frame may include generating the merge frame by taking an average value of data existing at the same position of each of the at least one preliminary frame among the plurality of preliminary frames, and applying the average value to the corresponding position of the merge frame. The location where no data exists in all of the plurality of preliminary frames may include setting a corresponding location of the merged frame as a closed section.

The generating of the final interpolation frame may include comparing the number of pixels of the occlusion period included in the merge frame with a preset reference value and performing intra-screen interpolation when the number of pixels of the occlusion period is smaller than the reference value. And interpolating the occlusion section.

Here, in the interpolation of the occlusion section by performing the interpolation, the occlusion section may be interpolated by taking an average value of pixels surrounding the occlusion section.

The generating of the final interpolation frame may include setting a specific number of pixel lines among pixels adjacent to the block including the occlusion section as a reference line when the number of pixels in the occlusion section is greater than or equal to the reference value. Rescanning a block including the occlusion section and a region including the set reference line in one frame and the second frame, and taking the average of blocks selected through rescanning in the first frame and the second frame; The interpolation may be performed by applying the block including the occlusion section.

The rescanning of the block including the occlusion section and the region including the set reference line in the first frame and the second frame may include generating a motion vector predicted in the generating of the plurality of preliminary frames. It is available.

Here, the rescanning of the block including the block section and the set reference line in the first frame and the second frame may be performed at least within a search region of the first frame and the second frame. A block having a sum of absolute histogram difference (SHAD) may be searched.

According to the frame interpolation method according to the present invention as described above, two preliminary frames are generated by performing block unit frame interpolation and motion vector unit frame interpolation with respect to a temporally continuous previous frame and a current frame, respectively. One merge frame is generated by merging the preliminary frames, and the final interpolated frame is generated by performing interpolation or rescanning on the screen according to the size of the occlusion section included in the generated merge frame.

Therefore, it is possible to improve the accuracy of the interpolated frame, thereby improving the objective and subjective image quality of the interpolated image.

1 is a flowchart illustrating a frame interpolation method according to an embodiment of the present invention.
2 illustrates a preliminary frame generated according to whether a threshold is applied in a frame interpolation method according to an embodiment of the present invention.
3 is a conceptual diagram illustrating a block position based preliminary frame generation process according to an embodiment of the present invention.
FIG. 4 illustrates two spare frames generated through the block position based preliminary frame generation process shown in FIG. 3.
5 is a conceptual diagram illustrating a process of generating a preliminary frame based on motion vector position according to an embodiment of the present invention.
FIG. 6 shows two spare frames generated through the motion vector position based preliminary frame generation process shown in FIG.
7 is a conceptual diagram illustrating a frame merging process according to an embodiment of the present invention.
FIG. 8 illustrates a merge frame generated through the frame merging process shown in FIG. 7.
9 is a conceptual diagram illustrating an intrapolation method of a merge frame in a frame interpolation method according to an embodiment of the present invention.
FIG. 10 illustrates a rescan target block for interpolating a closed section of a merged frame in a frame interpolation process according to an embodiment of the present invention.
FIG. 11 is a conceptual diagram illustrating a process of adaptively selecting a reference line used for rescan in a frame interpolation process according to an embodiment of the present invention.
12 illustrates a comparison between an image generated through intrapolation and an original image in a frame interpolation process according to an embodiment of the present invention.
13 is a conceptual diagram illustrating a block rescan process in a frame interpolation process according to an embodiment of the present invention.
14 shows an objective performance evaluation result of the frame interpolation method according to an embodiment of the present invention.
15 shows a subjective performance evaluation result of the frame interpolation method according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a frame interpolation method according to an embodiment of the present invention. The frame interpolation method shown in FIG. 1 may be performed in a frame interpolation apparatus. For example, the frame interpolation method according to an embodiment of the present invention may be performed by a frame interpolation module or chip implemented in hardware in various display apparatuses or may be performed in software.

Hereinafter, the overall operation of the frame interpolation method according to an embodiment of the present invention will be described with reference to FIG. 1.

The frame interpolation method according to an embodiment of the present invention comprises the steps of generating a plurality of preliminary frames by using block interpolation and motion vector unit interpolation (step 110) and merging the generated plurality of preliminary frames. And generating a final interpolated frame by correcting the occlusion period included in the generated merged frame (step 150).

First, in the preliminary frame generation process (step 110), a global search is performed using an interpolated image of 1/4 pixel units for accurate estimation of the motion vector (step 111). Motion compensation using subpixel units of integer pixel or less provides better compression efficiency, although the complexity is increased than that of integer pixel units. In addition, since the accuracy of the motion compensation in the unit of 1/4 pixel is better than the motion compensation in the unit of 1/2 pixel, H.264 / AVC stipulates to perform the motion compensation in unit of 1/4 pixel.

As described above, even when the motion vector is predicted using the interpolated image in the unit of 1/4 pixel, when the zooming in, zooming out, panning, light change, noise, etc. occur in the image, the wrong motion vector may be predicted. Can be.

Therefore, in the frame interpolation method according to an embodiment of the present invention, a threshold value T is set to select only a motion vector accurately predicted in the motion compensation prediction process and use it for frame interpolation (step 113).

In detail, since the referenced previous frame and the current frame are adjacent in time, the correlation between the two frames is high. Therefore, a threshold value is set using a sum of square difference (SSD), which is a method of calculating the sum of squared prediction error values of two frames. Equation 1 represents an SSD.

In Equation 1, N represents a block size, f _t-1 represents a previous frame, f _{t + 1} represents a current frame, and (i, j) represents a position of a pixel. Since the SSD considers the correlation between the reference block and the prediction block based on the error values of the pixels within the predicted size (N ㅧ N), each block has a different SSD value. Therefore, if the SSD is set to a threshold value that is optimal, it is possible to set a threshold value that can correspond to the change of the image.

On the other hand, PSNR (Peak Signal to Noise Ratio) is used as a measure of how much image quality deteriorates when compressing an image. The PSNR is used to measure an encoding error with an image reconstructed by the decoder based on an image input to the encoder. If the PSNR is small, the degradation of image quality is severe. If the PSNR is large, the subjective quality is good. As described above, when the PSNR is used to measure the deterioration of the image quality, a threshold value that is a reference for the prediction error may be set. Equation 2 shows a process of obtaining an SSD from the PSNR.

)를 곱하여 산출된다. In Equation 2, Mean Square Error (MSE) represents an average of difference values between an input image of an encoder and an image reconstructed from a decoder. An SSD value of a general block may be obtained by calculating a prediction error of each block from a PSNR value in which image degradation does not occur subjectively. Finally, the threshold value T is a scale factor 1/1 for comparison with a sum of absorptive difference (SAD) used to estimate a motion vector in a motion compensation process as shown in Equation 3 below.

Calculated by multiplying

As described above, when the threshold value T is set, in the global search process, the SAD is calculated between the prediction block and the candidate blocks, the candidate block having the minimum SAD is set as the reference block (115), and the SAD is determined in step 113. Compare with the threshold set in step 117. In this case, if the SAD is greater than or equal to the threshold T, the motion vector is inaccurate. Therefore, the block corresponding to the prediction block is excluded from the preliminary frame to be set as the occlusion interval, and the SAD is smaller than the threshold T. Only interpolate a block corresponding to the prediction block in the preliminary frame.

The preliminary frame generates two preliminary frames through block position based frame interpolation (step 119), and generates two preliminary frames through motion vector position based frame interpolation (step 121). Create a frame.

Subsequently, one merge frame is generated by merging a total of four preliminary frames generated in the preliminary frame generation process (step 130), and a final interpolated frame is generated by interpolating the occlusion sections included in the generated merge frame (step 150). ).

The process of generating the final interpolation frame (step 150) compares the number of pixels in the occlusion section with a preset threshold (β) (step 151), and in-screen interpolation when the number of pixels in the occlusion section is smaller than the threshold (β). To interpolate the occlusion section (step 153).

Alternatively, when the number of pixels in the occlusion section is greater than or equal to the threshold β, the block is selected to adaptively select a reference line corresponding to the occlusion section and then rescan using the occlusion section including the reference line to have a minimum SHAD. Is selected as a reference block, and then a final interpolation frame is generated by interpolating the occlusion interval using the selected reference block (step 155).

Hereinafter, a frame interpolation method according to an embodiment of the present invention shown in FIG. 1 will be described in more detail.

2 illustrates a preliminary frame generated according to whether a threshold is applied in a frame interpolation method according to an embodiment of the present invention.

FIG. 2A illustrates an image in which a threshold is not applied while generating a frame using a motion vector. Looking at the enlarged image, it can be seen that the tooth part of a human is generated from a block generated from an inaccurate motion vector, causing the image to break.

FIG. 2B shows a result of applying a threshold value to the same image. In FIG. 2B, a block predicting an incorrect motion vector is shown as a blockage section in FIG. 2B.

As illustrated in FIG. 2, in the frame interpolation method, only a block having high reliability is selected by applying a threshold in the process of generating a preliminary frame using a motion vector.

As described above, when the threshold value T is determined, a preliminary frame is generated using an image interpolated in units of 1/4 pixels. Here, the global search process is performed to detect the reference block having the minimum SAD value. SAD is used for efficient block prediction by calculating and summing the absolute values of the difference values between the reference block and the pixels included in the current block.

However, even if the block has the minimum SAD, a block containing an error may be selected. In order to determine a block containing such an error, in the process of generating a frame through bidirectional motion estimation, a threshold value T is applied to select only blocks having a SAD less than the threshold value T.

In the past, a block interpolation or motion vector position based frame interpolation method was selectively used to generate an interpolation frame using bidirectional motion estimation. However, in case of using only one method by setting the threshold value, since the occlusion section exists in each frame, the amount of data required for the additional interpolation process is insufficient.

Therefore, in the frame interpolation method according to an embodiment of the present invention, a block position based frame interpolation method and a motion vector position based frame interpolation method are applied by using the motion vector predicted in the motion compensation process, and then, a total of 4 operations are performed. Create a spare frame for the chapter.

Equation 4 represents the SAD calculation in the bidirectional motion prediction process, and equations 5 and 6 represent the frame interpolation process to which the threshold is applied.

In Equations 4 to 6, mv represents a motion vector, MCFI ^co represents a block position based frame interpolation method, and MCFI ^tr represents a motion vector position based frame interpolation method. Also, B _{x, y} represents the motion vector predicted from the reference frame f _t-1 (ie, the previous frame), and F _{x, y} represents the motion vector predicted from the reference frame f _{t + 1} (ie, the current frame). Indicates. That is, different motion vectors are predicted through a bidirectional motion compensation process of two reference frames. In Equation 5, α represents a threshold application weight. That is, in the block position based frame interpolation method, a lower threshold is applied than the motion vector position based frame interpolation method.

과, 수학식 6과 같이 움직임 벡터 위치 기반으로 보간된 두 장의 예비 프레임

을 생성한다.
As shown in Equation 5, by comparing the SAD of each block with the threshold value T, two preliminary frames interpolated based on the block position for the block having the SAD less than the threshold value T.

And two spare frames interpolated based on the motion vector position as shown in Equation (6).

.

3 is a conceptual diagram illustrating a block position based preliminary frame generation process according to an embodiment of the present invention, and FIG. 4 illustrates two preliminary frames generated through the block position based preliminary frame generation process shown in FIG. 3. .

및

)을 생성한다. 3 and 4, the block position based preliminary frame generation process includes two preliminary frames (2) through bidirectional block position based interpolation.

And

).

의 동일한 위치 (i, j)에 블록을 보간한다. First, as shown in (a) of FIG. 3, a forward frame is performed using a motion vector predicted in a previous frame f _{t −1} to perform a forward motion compensation.

Interpolate the block at the same position (i, j).

에 블록을 생성한다. In addition, as shown in (b) of FIG. 3, the backward motion compensation is performed to reserve the frame using the motion vector predicted from the current frame f _{t +1} .

Create a block in.

및

는 수학식 7과 같이 나타낼 수 있다.The last frame created

And

May be expressed as in Equation 7.

는 순방향 움직임 보상 과정으로부터 예측된 움직임 벡터 (B_x, B_y)가 적용된 이전 프레임 f_{t -1} 및 현재 프레임 f_{t +1}에 포함된 각 해당 블록의 평균값을 통해 생성된 프레임을 의미하고,

는 역방향 움직임 보상 과정으로부터 예측된 움직임 벡터 (F_x, F_y)가 적용된 이전 프레임 f_{t -1} 및 현재 프레임 f_{t +1}에 포함된 각 해당 블록의 평균값을 통해 생성된 프레임을 의미한다.In equation (7)

Denotes a frame generated through the average value of each corresponding block included in the previous frame f _{t -1} and the current frame f _{t +1} to which the motion vector (B _x , B _y ) predicted from the forward motion compensation process is applied.

Denotes a frame generated through an average value of each corresponding block included in the previous frame f _{t -1} and the current frame f _{t +1} to which the motion vectors F _x and F _y predicted from the backward motion compensation process are applied.

As shown in FIG. 4, the frame interpolation method generates two preliminary frames through a bidirectional block position based preliminary frame generation process.

)을 나타내고, 도 4의 (b)는 역방향 움직임 보상 과정을 수행하여 생성한 예비 프레임(

)을 나타낸다. 블록 위치 기반 프레임 보간 과정(MCFI^co) 에서는 수학식 5에 나타낸 바와 같이 움직임 벡터 위치 기반 프레임 보간 방법(MCFI^tr은) 보다 높은 임계값을 적용하였기 때문에 도 4에 도시한 바와 같이 생성된 프레임이 많은 폐색 구간을 포함하게 된다. 이는 프레임 병합 과정에서 발생할 수 있는 불필요한 영상의 중복을 방지하고, 최종적으로 생성된 보간 영상의 정확성을 높일 수 있다.
4 (a) shows a preliminary frame generated by performing a forward motion compensation process (

4, (b) shows a preliminary frame generated by performing a backward motion compensation process (

). In the block position based frame interpolation process (MCFI ^co ), as shown in Equation 5, since a higher threshold value is applied than the motion vector position based frame interpolation method (MCFI ^tr ), as shown in FIG. It will include an occlusion section. This may prevent duplication of unnecessary images that may occur in the frame merging process and increase the accuracy of the finally generated interpolated image.

FIG. 5 is a conceptual diagram illustrating a motion vector position based preliminary frame generation process according to an embodiment of the present invention, and FIG. 6 is two preliminary frames generated through the motion vector position based preliminary frame generation process shown in FIG. 5. Indicates.

및

)을 생성한다.5 and 6, the process of generating a motion vector position based preliminary frame includes two preliminary frames (2) through bidirectional motion vector based interpolation.

And

).

First, as shown in (a) of FIG. 5, the current block position (i, j) of the current frame f _{t + 1} is obtained using the motion vectors B _x and B _y predicted in the previous frame f _t-1 . An interpolation block is expressed at the intermediate position (i + 0.5B _x , j + 0.5B _y ) of the reference block (i + B _x , j + B _y ) of the previous frame f _t-1 by using Equation 8 below. Create

Also, as shown in (b) of FIG. 5, the current block position (i, j) of the previous frame f _{t-1 and} the current block f _t-1 using the motion vectors F _x and F _y predicted from the current frame f _{t + 1} An interpolation block is expressed in the intermediate position (i + 0.5F _x , j + 0.5F _y ) of the reference block (i + F _x , j + F _y ) of the current frame f _{t + 1} using Equation 9 below. Create

는 순방향 움직임 보상 과정으로부터 예측된 움직임 벡터 (B_x, B_y)의 중간 위치에 현재 프레임의 블록과 이전 프레임의 블록의 평균을 적용함으로써 생성한 예비 프레임을 나타내고,

는 역방향 움직임 보상 과정으로부터 예측된 움직임 벡터 (F_x, F_y)의 중간 위치에 현재 프레임의 블록과 이전 프레임의 블록의 평균을 적용함으로써 생성한 예비 프레임을 나타낸다. here,

Denotes a preliminary frame generated by applying an average of blocks of the current frame and blocks of the previous frame to intermediate positions of the motion vectors (B _x , B _y ) predicted from the forward motion compensation process,

Denotes a preliminary frame generated by applying an average of a block of a current frame and a block of a previous frame to an intermediate position of a motion vector (F _x , F _y ) predicted from a backward motion compensation process.

When generating the preliminary frame based on the predicted motion vector as shown in FIG. 5, each block constituting the preliminary frame is irregularly positioned according to the motion vector. That is, in the preliminary frame generated as shown in FIG. 6, a section overlapping between blocks may occur, or a blockage section in which an image is not generated may occur. Here, data loss can be prevented by taking an average value of two areas in areas where blocks overlap.

In addition, since the two preliminary frames generated as shown in FIG. 6 consider different motion information, positions of the occlusion sections may be different even in the same image.

As described above, when two preliminary frames are generated through block position based frame interpolation, and two preliminary frames are generated through motion vector position based frame interpolation, a total of four preliminary frames generated are one for processing the occlusion interval. Merge into frames. Here, each generated preliminary frame may include occlusion sections at different positions because the interpolation process and the applied motion vector are different.

7 is a conceptual diagram illustrating a frame merging process according to an embodiment of the present invention, and FIG. 8 illustrates a merge frame generated by merging respective preliminary frames.

)을 생성한다. 또는, 모든 프레임에 데이터가 존재하지 않는 경우에는 병합 프레임의 해당 위치를 폐색 구간으로 선택한다.7 and 8, in the frame merging process, when data exists at the same position of all four preliminary frames, the average value of the data at the same position of all the frames is taken and applied to the corresponding position of the merged frame. If no data exists in the above spare frame, the average value is taken for the data in the same position of the remaining frames except for the preliminary frame in which no data exists, and then applied to the corresponding position of the merge frame.

). Alternatively, when data does not exist in all frames, the corresponding position of the merge frame is selected as the occlusion section.

)의 해당 구간도 역시 폐색 구간으로 남게 되는데, 이와 같은 폐색 구간에 대해서는 화면내 보간 방법 또는 재탐색을 통한 움직임 벡터의 예측을 이용하여 영상을 보간하여 최종 보간 영상을 생성한다.
As described above, in the process of merging four preliminary frames, many occlusion sections included in each preliminary frame may be interpolated. However, for a section in which no data exists in all spare frames, as shown in FIG. 8, the merge frame (

The corresponding section of) also remains as an occlusion section. For this occlusion section, the final interpolation image is generated by interpolating the image using an intra-screen interpolation method or a prediction of a motion vector through rescanning.

9 is a conceptual diagram illustrating an intrapolation method of a merge frame in a frame interpolation method according to an embodiment of the present invention.

)에는 화소 형태의 폐색 구간이 존재할 수 있고, 이와 같은 화소 형태의 폐색 구간은 그 영역이 작기 때문에 움직임 보상을 통한 보간 보다 화면내 보간을 수행하는 것이 연산량을 줄이면서 정확도를 향상시킬 수 있다. 수학식 10은 병합 프레임(

)에 포함된 화소 단위의 폐색 구간의 화면내 보간을 수행하기 위한 수식을 나타낸다.As shown in FIG. 8A, the merge frame (

), There may be a pixel occlusion section, and since the pixel occlusion section has a small area, performing interpolation on the screen rather than interpolation through motion compensation may improve the accuracy while reducing the amount of computation. Equation 10 is a merge frame (

) Represents an expression for performing intra-screen interpolation of the occlusion section in units of pixels.

In Equation 10, β denotes a threshold value for determining a block for performing intra-screen interpolation, and an embodiment of the present invention will be described with reference to 16. In addition, S represents an average value of pixels surrounding the selected occlusion section, and N represents the number of pixels for which there is no neighboring data.

That is, as shown in (a) of FIG. 9, when the number of pixels not present in the predetermined block is smaller than β (ie, 16), the interval is interpolated using neighboring data as shown in Equation 10. .

9B shows an image of the result of performing the interpolation in the screen by the above-described method. As shown in (b) of FIG. 9, the interpolation interval included in the merged frame in which the interpolation is performed may generate the final interpolation image by performing interpolation using prediction of the motion vector through rescan.

FIG. 10 illustrates a rescan target block for interpolating a closed section of a merged frame in a frame interpolation process according to an embodiment of the present invention, and FIG. 11 is used for rescan in a frame interpolation process according to an embodiment of the present invention. A conceptual diagram illustrating a process of adaptively selecting a reference line to be used. 12 illustrates a comparison of an image generated by interpolation and an original image in a frame interpolation process according to an embodiment of the present invention, and FIG. 13 illustrates a block in a frame interpolation process according to an embodiment of the present invention. A conceptual diagram illustrating the rescan process.

Hereinafter, referring to FIGS. 10 to 13, a process of interpolating a closed section included in a merge frame through rescan is described.

)에 포함된 폐색 구간을 보간하는 과정에서는 병합 프레임(

)을 이용하여 움직임 보상을 수행한다. 즉, 재탐색 과정에서는 폐색 구간이 포함된 블록을 공간적으로 둘러싸고 있는 화소들의 데이터를 이용하여 보다 정확한 움직임 벡터를 예측한다. In the preliminary frame generation process described with reference to FIGS. 3 to 6, motion compensation is performed using two temporally neighboring frames f _t-1 and f _{t + 1} to generate a frame that does not exist. However, merge frames created by merging four spare frames (

), Interpolating the occlusion sections in

) To perform motion compensation. That is, in the rescan process, a more accurate motion vector is predicted using data of pixels spatially surrounding the block including the occlusion section.

Since the motion compensation is performed on a block-by-block basis, there are pixel data available in the block even in the block section as shown in FIG. 10. Accordingly, the frame interpolation method according to an embodiment of the present invention adaptively selects the neighboring pixel information of the block together with the pixel data existing in the block including the occlusion period in the merge frame and performs the rescan process.

Hereinafter, a process of adaptively selecting a reference line used for rescanning will be described in more detail with reference to FIG. 11.

First, FIG. 11A illustrates a case in which all of the neighboring pixel data of the predetermined block can be used together with the pixel data existing in the block including the occlusion section. In such a case, the referenced line is set to two. That is, as shown in (a) of FIG. 11, when the occlusion section exists only in one block and there is no occlusion section in a block adjacent to the block, two of the pixels adjacent to the block including the occlusion section are present. Pixel lines of lines are included and used for rescanning.

However, as shown in (b) to (e) of FIG. 11, when a block section also exists in a block including a block section and a neighboring block, it can be used for rescan compared to FIG. 11 (a). Since the pixel data is small, a reference line is added to correspond to the increase in the occlusion interval to compensate for this.

In FIG. 11 (b), the reference line is set to 3, in FIG. 11 (c), the reference line is set to 4, in FIG. 11 (d), the reference line is set to 6, and in FIG. ) Shows a case where the reference line is set to 5. However, the number of reference lines set according to the size of the occlusion section is not limited to the value shown in FIG. 11 but may be set to a value optimized for the size of the occlusion section through repeated experiments.

As described above, when the reference line is adaptively selected according to the size of the occlusion period, a final rescan process is performed on the previous frame and the current frame by using the pixel data existing in the selected reference line and the block. In the rescan process, since the motion vector predicted in the initial search process can be used, the search area can be reduced.

In the global search process for predicting the motion vector in generating the preliminary frame, the error vector of the two blocks is calculated using SAD to predict the motion vector of the block having the minimum SAD. In this case, the SAD value could be calculated accurately because two images are actually compared.

However, in the rescanning process, since the motion vector is estimated using a virtual image (that is, a merged frame) generated using the preliminary frames, an accurate block cannot be predicted using the SAD. That is, as shown in FIG. 12, it is difficult to find a difference between the two images in the case of the subjective image quality of the entire image and the original image. However, when looking at the enlarged specific area of the image, a slight difference in pixel value can be confirmed. This result is because the image generated through the motion compensation process is subjective almost similar image, but the image is not completely matched. Therefore, when the SAD is calculated when the image information having the minute difference as described above is used, a problem of predicting a wrong motion vector occurs rather than increasing the range of the error.

Therefore, in the frame interpolation method according to an embodiment of the present invention, a method of predicting a motion vector by analyzing the overall propensity of pixel data without considering the minute error of the interpolated frame is SAHD (Sum of Absolute Histogram Difference). Apply.

The SAHD sets a bin for each pixel value as shown in Equation 11, and then selects a block having a minimum difference value by calculating a cumulative value of a corresponding section.

및

는 각각 보간 프레임과 참조되는 프레임의 빈 내에 포함되는 히스토그램 누적치를 나타낸다. 탐색 영역 내에서 최소 SHAD를 가지는 블록을 탐색하여 수학식 12를 이용하여 최종 보간 프레임

를 생성한다.In Equation 11, b represents a set bin,

And

Represents a histogram cumulative value included in bins of the interpolated frame and the referenced frame, respectively. Search for the block with the smallest SHAD in the search area and use the equation (12) to final interpolate frame

.

및

는 각각 참조 라인으로 선택된 화소 정보를 이용하여 재탐색 과정을 수행하여 예측된 움직임 벡터를 나타낸다.In Equation 12, B _{x, y} and F _{x, y} represent a motion vector obtained from an initial global search process for generating a preliminary frame,

And

Represents a predicted motion vector by performing a rescan process using pixel information selected as reference lines, respectively.

및

는 순방향 및 역방향의 참조 프레임에 따라 독립적으로 적용되고, 이에 따라 각 참조 프레임에서 선택된 블록에 대한 평균값을 취함으로써 최종적인 보간 프레임

을 생성한다.
Bar and motion vector shown in FIG.

And

Are independently applied according to the forward and backward reference frames, and thus the final interpolated frame by taking the average value for the selected block in each reference frame.

.

14 shows an objective performance evaluation result of the frame interpolation method according to an embodiment of the present invention.

In the performance evaluation, the PSNRs of the frame interpolation method and the conventional Motion Compensated Frame Interpolation (MCFI) method are compared and analyzed according to an embodiment of the present invention.

Foreman, Bus, Flower, Mobile, Tempete, Akiyo, Children, City, Coastguard are all CIF (352 × 288) size, and the size of block used for motion compensation prediction is 8 × 8. The search area in the step for initial MCFI selection was searched up to 1/4 pixels at 16 × 16.

In addition, the search area in the step of predicting the motion vector using the pixel information included in the neighboring block is set to 2 × 2 because the median value of the motion vector of the neighboring block is considered.

As shown in Figure 14, it can be seen that the PSNR of the frame interpolation method according to an embodiment of the present invention has a more improved performance than the conventional method. The frame averaging (FA) method has a low PSNR value because it calculates an average of spatially identical blocks of two neighboring frames. In addition, since MCFI performs a motion compensation process, it shows an improved result than the frame averaging (FA) method, but there are also many problems when comparing subjective picture quality. On the other hand, the frame interpolation method according to an embodiment of the present invention has an improved PSNR compared to conventional frame interpolation methods. For example, in the case of mobile video, PSNR is improved by 2.96dB compared to MCFI, foreman video is about 2.46dB, and flower video is improved by 1.77dB.

15 shows a subjective performance evaluation result of the frame interpolation method according to an embodiment of the present invention.

The subjective performance evaluation compared the subjective picture quality of the conventional frame interpolation method (FA, MCFI) and the frame interpolation method according to an embodiment of the present invention for the Foreman image.

FIG. 15 (a) shows an original Foreman image, and FIG. 15 (b) shows a frame interpolation result using a conventional frame averaging (FA) method, since it is an image generated by an average of two neighboring frames. The result is that two frames overlap.

FIG. 15C illustrates a conventional Motion Compensation Frame Interpolation (MCFI) result and does not accurately interpolate a face portion where a motion occurs.

However, as shown in (d) of FIG. 15, the frame interpolation method according to the embodiment of the present invention can be confirmed that the blocking effect is reduced compared to the conventional frame interpolation method, thereby improving the subjective picture quality.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

Claims (11)

Generating a plurality of preliminary frames through block position based frame interpolation and motion vector position based frame interpolation using the temporally successive first and second frames;
Merging the generated plurality of preliminary frames to generate one merge frame; And
And generating a final interpolation frame by interpolating the occluded sections included in the merged frame. The method of claim 1, wherein the generating of the plurality of spare frames comprises:
Searching for a reference block corresponding to the current block of the second frame through a global search in the first frame;
Setting a threshold value based on a peak signal to noise ratio (PSNR);
Comparing the threshold and the SAD of the reference block searched through the global search; And
The interpolation block is generated through the block position based frame interpolation and the motion vector based frame interpolation for the current block corresponding to the reference block smaller than the threshold and the reference block smaller than the threshold. And generating the plurality of preliminary frames. The method of claim 2, wherein the generating of the plurality of spare frames
And generating two preliminary frames through the bidirectional block position based frame interpolation, and generating two preliminary frames through the bidirectional motion vector based frame interpolation. The method of claim 2, wherein the generating of the plurality of spare frames
And applying the weight of the threshold value to set the block position based frame interpolation to have a lower threshold value than the motion vector based frame interpolation. The method of claim 2, wherein setting a threshold value based on the PSNR comprises:
Calculating a sum of square difference (SSD) by calculating an error of each block from the PSNR value; And
And applying the correction coefficient to the SSD to compare the SSD with a sum of absolute difference (SAD) to set the threshold value. The method of claim 1, wherein generating the one merge frame
The merge frame is generated by taking an average value of data existing in the same position of each of the at least one preliminary frame among the plurality of preliminary frames and applying the same to the corresponding position of the merge frame, and no data exists in all of the plurality of preliminary frames. Where not set the frame interpolation method, characterized in that for setting the corresponding position of the merge frame to the occluded section. The method of claim 1, wherein generating the final interpolation frame,
Comparing the number of pixels of the occlusion period included in the merge frame with a preset reference value; And
And interpolating the occlusion section by performing interpolation on the screen when the number of pixels in the occlusion section is smaller than the reference value. The method of claim 7, wherein the interpolating the occlusion section by performing the interpolation of the screen comprises:
And interpolating the occlusion section by taking an average value of pixels surrounding the occlusion section. The method of claim 7, wherein generating the final interpolation frame,
Setting a specific number of pixel lines among pixels adjacent to the block including the occlusion section as a reference line when the number of pixels in the occlusion section is greater than or equal to the reference value;
Rescanning a block including the occlusion section and an area including the set reference line in the first frame and the second frame; And
And interpolating the occlusion section by taking an average of the blocks selected through rescanning in the first frame and the second frame and applying the block to the block including the occlusion section. The method of claim 9, wherein the rescanning of the block including the occlusion section and the region including the set reference line in the first frame and the second frame comprises:
And a motion vector predicted in the step of generating the plurality of preliminary frames. The method of claim 9, wherein the rescanning of the block including the occlusion section and the region including the set reference line in the first frame and the second frame comprises:
And searching for a block having a minimum sum of absolute histogram difference (SHAD) in a search region of the first frame and the second frame.

Images