KR100969420B1 - Frame rate conversion method - Google Patents

Abstract

The present invention relates to a frame rate conversion method for converting a low frame rate video into a high frame rate video by increasing its frame rate. The frame rate conversion method according to the present invention includes predicting motion information for a previous frame and a current frame and generating a new frame located between the previous frame and the current frame by applying the predicted motion information. . As a result, it is possible to prevent deterioration of the image quality in displaying a high frame rate moving image, so that it is possible to view a moving image having more improved image quality quality.

Frame rate, transform, motion vector, search window, block

Description

FRAME RATE CONVERSION METHOD

The present invention relates to a frame rate conversion method, and more particularly, to a frame rate conversion method for converting a low frame rate video into a high frame rate video.

Factors influencing consumers' purchasing decisions in video output devices such as mobile phone projection displays include price, size, design, lifespan, power consumption and picture quality. Among these factors, image quality is one of the major factors that can differentiate the products of competitors or late comers. Color image signal processing technology is one of the factors that determine the quality of image to be differentiated, and the importance of the image is emphasized in recent years. In this respect, the color image signal processing technology for improving the image quality is considered one of the most important technology for the mobile phone projection display.

Among the color image processing technologies, a frame conversion technique may be applied to improve the image quality of a mobile phone projection display. The frame conversion technology refers to a technique of increasing the frame rate of an input video without specific artifacts when converting a low frame rate input video into a high frame rate output video. 1 illustrates an example of converting a low frame rate input video into a high frame rate output video by applying a frame conversion technology of a frame rate conversion technology. For example, when a DMB image of 30 frames / sec is used as an input of a mobile phone projection display, problems such as motion judder and motion blurring occur. In order to solve such a problem, a technique for converting 30 frames / second to 60 frames / second or more frames is required.

Conventional techniques for frame rate conversion can be broadly divided into two categories depending on whether motion information of an object is used. There is a frame repetition and a frame average in the method of not using the motion information of the object, and motion compensated interpolation in the method of using the motion information.

2 illustrates an example of a position of a moving object generated by a conventional frame repetition method. If the observer is watching as the subject moves, the position expected by the observer will be different from the subject's position as seen by frame repetition. The position of the object expected from the observer is indicated in the fourth frame position of FIG. 2. The frame repetition method shows a good performance in an area without motion, but causes a problem such as motion judder in an area without motion, which causes a large deterioration in image quality.

3 illustrates an example of a position of a moving object generated by a conventional frame averaging method. Frame averaging is a method of averaging two neighboring frames to obtain an interpolated output frame. Like the frame repetition method, this method shows good performance in the region without motion. However, motion blurring occurs more severely in the moving region than the result of the frame repetition method.

Since the motion information is not used in the conventional method as described above, artifacts such as motion judder and motion blurring occur in the moving region of the frame. Motion compensation interpolation method has been developed to solve this problem and to place the moving object in the correct position. The motion compensation interpolation method includes predicting a motion vector and interpolating a frame using the predicted motion vector. 4 shows an example of an output frame interpolated by a motion compensation average of neighboring frames. If the motion vector is accurately predicted, no artifacts, such as motion blur, occur in the moving region. However, a technique for accurately predicting motion vectors is difficult to implement. If the predicted motion vector is incorrect, problems such as motion blur and block artifacts occur in the moving region. In addition, since the degree of accuracy of the predicted motion vector depends on the speed or size of the object, more severe image quality degradation may occur than motion blur caused by frame repetition.

Conventional motion vector prediction methods include a pixel recursive algorithm that predicts motion on a pixel-by-pixel basis and a block matching algorithm that predicts motion information on the assumption that they have the same motion on a block-by-block basis. In many video codings, block matching algorithms are widely used in consideration of data flow regularity, computational complexity, and hardware implementation. 5 is an example illustrating a process of predicting a motion vector for a block of size X * Y in a search region of a reference frame. The block matching algorithm is an algorithm that finds a block having the smallest matching error in a search window in units of blocks and estimates horizontal and vertical displacement of the block. The block matching algorithm divides one frame of a video into several blocks and finds the block having the smallest matching error in the search window of a reference frame for each block thereof. At this time, the position difference between the block to be processed and the block that is best matched in the reference frame is called a motion vector.

Among the widely used Cost functions used to calculate the matching error are the summed absolute difference (SAD), the summed square error (SSE), and the normalize cross correlation function (NCCF). SAD is calculated using [Equation 1].

는 후보 움직임 벡터의 좌표값이고,

는 화소의 좌표값을 나타낸다.

는 현재 처리하고자 하는 블록의 중심 좌표값을 나타내고,

는

를 중심으로 하는 블록을 나타낸다.

는 t 번째 프레임의

에 해당하는 화소의 밝기값이다. 그리고 t는 시간을 나타내는 변수이고, n 과 T는 자연수이다.here,

Is the coordinate value of the candidate motion vector,

Represents the coordinate value of the pixel.

Represents the center coordinates of the block to be processed.

Is

Represents a block centered on.

Of the t th frame

The brightness value of the pixel corresponding to. T is a time variable, and n and T are natural numbers.

SSE and NCCF are calculated by Equation 2 and Equation 3, respectively.

SAD is the simplest to implement. Because SSE is a squared operation, a large number of bits are needed to represent a range of error values. Therefore, SSE calculation requires more hardware than SAD. Since NCCF performs multiplication and division operations, complex hardware is required. Therefore, SAD is widely used for real time processing.

Conventional methods of searching for motion vectors include various methods such as full search, subsampling search, three-step search, two-dimensional logarithm search, and diamond search. Hereinafter, a conventional search method will be described in detail with reference to the accompanying drawings.

Global search is the simplest block matching method. Global search predicts the motion vector after calculating the matching error for all possible candidate motion vectors within the search range. This method is computationally expensive because it calculates the matching error for all candidate motion vectors. Global search is thus difficult to use for real-time video coding applications and software implementations. In order to solve this problem, various fast block matching methods have been developed.

The subsampling search reduces the amount of computation by reducing the number of pixels used to calculate the matching error. Representative methods include alternating 4: 1 pixel sub-sampling, sub-sampled motion field estimation, and sub-block motion field estimation.

Alternating 4: 1 pixel sub-sampling uses four subsampling patterns. FIG. 6 is a diagram for explaining a subsampling pattern and shows 8 × 8 blocks labeled 'a', 'b', 'c', and 'd' in each pixel. Pattern A is a sub-sampling pattern composed of all 'a' pixels in an 8x8 block. Similarly, patterns B, C, and D are subsampling patterns composed of all the 'b', 'c', and 'd' pixels. In the conventional 4: 1 sub-sampling method, only pixels of the pattern A are used for block matching, and as a result, the amount of calculation is reduced to 1/4. However, since 3/4 pixels in an 8x8 block are not used for matching calculation, it is difficult to trust the accuracy of the predicted motion vector. The Alternating 4: 1 pixel sub-sampling technique developed to solve this problem uses all four subsampling patterns A, B, C, and D.

7A and 7B are diagrams illustrating a use order of four subsampling patterns on a search area. Each pixel is labeled '1', '2', '3', and '4'. When the first pixel (the upper-left position of the block) of the candidate block is indicated by 1, only pixels of the pattern A are used for calculating the matching error. In the case of FIG. 7A, after calculating the matching error for the pattern A, the pattern D is used to calculate the matching error for the candidate block located to the right of one pixel. For each block to be processed, four candidate blocks with minimum matching errors for each pattern A, B, C, and D will be selected. Using all the pixels, the matching error between the block to be processed and the four candidate blocks is obtained, and then the position of the block having the minimum matching error is selected as the coordinate of the final motion vector. Processing according to the order of use of the four subsampling patterns uses all the pixels in the search region of the previous frame.

The sub-sampled motion field method uses the characteristic that the motion of the video sequence is changed smoothly and slowly. Thus, the motion vectors of neighboring blocks are almost similar. Assuming that the motion vectors for the block to be processed and the neighboring blocks are similar to each other, first, a motion vector for half of the block in the current frame is predicted.

8 is a diagram illustrating a block on a current frame and illustrating a sub-sampled motion field method. In FIG. 8, four motion vectors for a dark block adjacent to the center block A are found, and a motion vector having a minimum matching error with the block A is selected as the motion vector of the block A. If the motion of block A is about the same as the direction and magnitude of the motion of the neighboring block, this method can successfully predict the motion vector. However, when block A includes two objects moving in different directions, it is difficult to predict an accurate motion vector for the motion of block A predicted by the sub-sampled motion field method.

When the size of a block used in the block matching method is N × N, sub-block motion field estimation uses a block of size N / 2 × N / 2. 9 is a diagram showing that each block is divided into four sub-blocks having a size of N / 2 × N / 2 in the current frame. In FIG. 9, A, a, b, and c represent subblocks within one block, and B, C, and D represent subblocks of neighboring blocks. First, the motion vectors of the sub-blocks A, B, C, and D in the upper left corner of each block are predicted. Then, the motion vector of the remaining subblock is predicted using the motion vector of the predicted neighboring subblock. For example, in FIG. 9, the motion vector of the subblock a is assigned to be better matched among the motion vectors of A and B. Further, the sub block b is selected from the motion vectors of A and C, and the sub block c is selected from the motion vectors of A, B, C, and D. However, since the sub-block motion field estimation method uses a smaller block size, there is a possibility of incorrectly predicting a motion vector in an area where there is little motion. Techniques in this category reduce the amount of memory used in hardware because only pixels limited by certain rules are used to calculate the matching error.

Three-stage search is widely used for video compression because of its simplicity and efficiency by examining several search points over a large area and then gradually narrowing them down. 10A to 10C are diagrams for describing a process of performing a three-stage search. When the search window is 7, the first step of the three-stage search finds the point with the minimum matching error using nine search points evenly allocated on a 7x7 grid as shown in FIG. 10A. . In the next step, as shown in FIG. 10B, eight search points are added around the search points of the minimum matching error found in the previous step. The distance between the two search points is then reduced by half that of the previous step. In the next step, as shown in FIG. 10C, the point having the minimum matching error is found while reducing the distance between the search points in the current step by half until the distance between the search points becomes 1. The position of the search point having the minimum matching error among the nine search points means a motion vector.

FIG. 11 is a diagram illustrating an error surface showing a matching error in a search area. FIG. As shown in FIG. 11, it can be seen that there are several local minimums in the search area. The first phase of the three-step search uses search points evenly allocated within the search area, making it easy to fall into local minimums. Therefore, the three-stage search method is inefficient when estimating small movements.

The two-dimensional logarithm search is a method of searching five search points on a 5 × 5 grid, repeatedly comparing the block matching error of each search point, and then searching by reducing the search range in half.

12A to 12C are diagrams for describing a process of performing a 2D logical rhythm search. As shown in FIG. 12A, when the search window is 7, the two-dimensional logarithmic search finds the point having the minimum matching error by using five search points evenly allocated on the 5 × 5 grid. As shown in FIG. 12C, when the search point having the minimum matching error is the same as the center of the five search points, eight search points are added around the search point having the minimum matching error. The distance between the two search points is half that of the initial stage. If the search point having the minimum matching error is not the same as the center of the five search points, the process of the previous step is repeated around the search point having the minimum matching error as shown in FIG. 12B. This process is repeated until the point of the minimum matching error is equal to the center on the 5x5 grid. The position of the search point having the minimum matching error among the nine search points becomes the coordinate of the motion vector.

Diamond search is performed under the assumption that the distribution of motion vectors is concentrated in the center of the search window. Diamond search uses two search patterns. 13A-13B illustrate two search patterns used in diamond search. The first search pattern is a large diamond search pattern as shown in FIG. 13A, which is diamond shaped, and the second search pattern is a small diamond search pattern having a smaller size, as shown in FIG. 13B. In the diamond search, the search is repeated using a large diamond search pattern until the minimum block matching error is at the center of the search pattern.

14 illustrates an example of a diamond search. As shown in FIG. 14, five search points are added when the position having the minimum matching error of the large diamond search pattern is present at the vertex of the diamond, and the position having the minimum matching error of the large diamond search pattern is the face portion of the diamond. If present, three search points are added. If the location with the minimum matching error of the large diamond search pattern is at the center of the diamond, the search pattern is changed from the large diamond search pattern to the small diamond search pattern. At this time, the position of the search point having the minimum block matching error in the small diamond search pattern is equal to the coordinate of the predicted motion vector.

In the motion prediction method, the accuracy of the predicted motion vector varies depending on the size of a block used. In this conventional method, there is a problem in that the accuracy of the predicted motion vector is not constant according to the size of the target because a fixed size block is used. For example, if two or more objects having different directions are included in one block, it is difficult to predict the exact motion vector of the block.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems of the prior art, and predicts a motion vector and uses a predicted motion vector while using a simple amount of computation and hardware to improve the image quality of a mobile phone projection display. It is an object of the present invention to provide a frame rate conversion method capable of improving the rate.

In addition, an object of the present invention is to provide a frame rate conversion method that can prevent artifacts such as motion judder or motion blur by using blocks and search areas of variable size according to the degree of movement of the object.

The frame rate conversion method according to the present invention for solving the above problems is a frame rate conversion method for converting a low frame rate video to a high frame rate video by increasing the frame rate, according to the speed or size of the target A first step of predicting a motion vector using blocks of different sizes and a search window, and a second step of converting a low frame rate video into a high frame rate video using the predicted motion vector.

The first step may include a first process of predicting a motion activity of a block of a first size at the same position as a previous frame and a current frame, and using the predicted motion activity to determine the size and speed of an object. A second process of determining an appropriate block and size of a search window and a third process of predicting a motion vector of each block using a block matching algorithm.

Here, the second process is a type determination process for determining whether a block is a static type or a dynamic type according to the SAD value, a block merging process for expanding the size of the block by merging two neighboring identical sized static type blocks and the block. A search window size determining process may be performed to determine a size of the search window differently according to the type of.

In addition, the second process repeats the block merging process until the block of the second size also becomes between the merged blocks.

The third process is a forward motion prediction process of determining a motion vector by finding a position having a minimum matching error with a block of the current frame in a search window of a previous frame, and a SAD value at each position in the search window of the previous frame and its If the difference between the minimum SAD value and the number of positions smaller than the first threshold is larger than the second threshold value, the backward direction of determining the motion vector by finding the position having the minimum matching error with the block of the previous frame in the search window of the current frame. If the minimum prediction error between the motion prediction process, the correction process for correcting the still motion, and the block to be processed on the current frame and the block predicted on the previous frame is smaller than the third threshold, the motion prediction is performed again on the extended search window. Includes a redo process.

The frame rate conversion method according to the present invention configured as described above results in minimizing artifacts such as motion blur or block artifacts by predicting a motion vector using a block size and a search window suitable for the speed and size of an object. In addition, the calculation amount can be reduced by using a small search window in the region where there is little movement.

That is, in applying the frame rate conversion method to an input video having a low frame rate, it is possible to improve image quality while using simple hardware and a small amount of calculation, so that it is possible to perform real-time video processing on a mobile phone projection display. There is.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention. However, in describing the present invention, when it is determined that the detailed description of the related known technology or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intention or custom of the user, the operator, and the like. Therefore, the definition should be made based on the contents throughout the specification.

15 is a flowchart illustrating the overall flow of performing the frame rate conversion method according to the present invention. As shown in FIG. 15, first, a previous frame and a current frame are input (S110), and a difference between a previous frame and a current frame is calculated (S120).

The case where the difference between the previous frame and the current frame is large, that is, when the screen is switched, etc., is called a transition type. The two frames are compared to compare whether they are a transition type or not (S130). .

If the frame is a transition type, the frame interpolation step is performed without going through the motion prediction step (S150). The quality of the resulting image is not good because the accuracy of the predicted motion vector in the transition type frame is low. If it is not a transition type, the proposed motion prediction method is performed (S140), and the frame interpolation step is performed using the predicted motion vector (S150). Equation 4 below is an equation representing a frame interpolation method to be performed in the case of a transition type.

Where α is a constant between 0 and 1, and n is the index of the frame.

16 is an overall flowchart of a motion prediction technique according to the present invention.

As shown in FIG. 16, the performing of the motion prediction technique according to the present embodiment may include inputting pixel brightness values of 16 8 × 8 blocks at the same position of the previous frame and the current frame (S141). In step S142, a motion activity level of sixteen 8x8 blocks is predicted as a brightness value, in which a size of a block and a search window is determined based on the predicted motion activity, in step S143. And a step of performing a block matching method using the size of the search window (S144) and storing a predicted motion vector for each block (S145).

The motion activity for the 8x8 block means the motion size for that block. The motion activity of the block is predicted by the SAD between blocks in the same position of the current frame and the previous frame, as shown in [Equation 5]. The type of the block is determined by the calculated SAD value. If the value of the calculated SAD is smaller than the predetermined threshold value, the block is determined to be a no-motion type with little motion, otherwise it is determined as a motion type with motion. The determined block type is used in the next step to size the block and the search window.

The block size is divided into block classes according to the block type determined in the previous step. 17A to 17G illustrate each block class and the size and type of blocks belonging thereto. For example, FIG. 17B (a) shows a second class, and FIG. 17B (b) shows the types of blocks belonging to the second class. In the types of blocks belonging to each class, the number 1 represents a no-motion type, and 2 represents a motion type. If neighboring blocks are of no-motion type, they are merged into larger blocks. For example, if all four 8x8 blocks are of no-motion type, four 8x8 blocks are merged into one 16x16 block. Block sizes used in the present invention are 8x8, 16x8, 8x16, 16x16, 32x32.

18 shows an example of a block size applied on an interpolated frame according to the present invention. The test image of FIG. 18A includes an object of a simple movement such as a rectangle moving by 2 pixels for each adjacent frame interval from left to right. The test image of FIG. 18B includes the object of complex movement. As shown in FIGS. 18A and 18B, it can be seen that 32x32 blocks are used in the background region where there is little movement, and smaller blocks are used in the movement region.

The size of the search window is determined by the type of block already determined. Specifically, the size of the search window is larger by ± 7 when the block type is a motion type, and the size of the search window is smaller by ± 2 when the block type is a no-motion type. The larger search window is used when the motion complexity of the block is large, whereas the size of the search window used is relatively small in the case of a no-motion type with little motion of the block. Therefore, according to the present invention, the amount of computation is reduced by using a different size of the search window according to the block type.

Next, a step of predicting a motion vector of each block using a block matching algorithm will be described. In the predicting the motion vector, the motion vector is predicted by applying the size of the block determined in the previous step and the size of the search window.

19 is a flowchart illustrating a process of predicting a motion vector of each block using a block matching algorithm. Referring to FIG. 19, first, a forward motion prediction step is performed (S210). In forward motion prediction, the previous frame is used as a reference frame. The matching error criterion used is SAD as shown in [Equation 6] and the motion vector is predicted using [Equation 7]. As shown in [Equation 7], the motion vector means a position having a minimum matching error in the search window.

은 n번째 프레임 상에 있는

를 중심으로 하는 블록에 대하여 예측된 움직임 벡터를 의미한다.here,

Is on the nth frame

Means the motion vector predicted for the block centered on.

과 최소 SAD값인

사이의 차이 값이 산출된다. 탐색 윈도우 상에 최소 SAD값과 유사한 SAD값을 갖는 위치 가 많은 경우, 탐색 윈도우 상의 계산된 SAD 분포에는 다수의 지역 최소값(local minimum)이 존재할 가능성이 크다. 이와 같은 문제점을 해결하기 위하여

와

의 차이 값이 미리 결정된 문턱값보다 큰 위치의 개수(블럭의 수)가 소정의 기준치 이상일 경우, 역방향 움직임 예측 단계가 수행된다(S220, S230, S240).Next, the SAD value between the block at each position in the search window and the current block

And the minimum SAD value

The difference value between is calculated. If there are many locations with similar SAD values on the search window, it is likely that there are multiple local minimums in the calculated SAD distribution on the search window. To solve this problem

Wow

If the number of positions (number of blocks) whose difference value is larger than a predetermined threshold value is greater than or equal to a predetermined reference value, backward motion prediction steps are performed (S220, S230, S240).

In the backward motion phase, the current frame is used as a reference frame. When applied to an object moving or stopping in one direction such as a caption, a large number of local minimums are often present in the calculated SAD distribution on the search window.

20A and 20B illustrate examples of subtitles moving at the same speed from right to left. Assuming that the block in the rectangular frame in FIG. 20A and FIG. 20B is the block to be processed at present, in the forward motion prediction, the block in the rectangular frame of FIG. 20B is used as a reference and a search is performed on the previous frame. However, since the blocks in the rectangular border of FIG. 20B include objects that were not on the previous frame, it is almost impossible to find a block that exactly matches the block to be processed on the previous frame. Accordingly, by performing backward motion prediction, a block that is exactly matched with a block to be processed may be determined.

과

의 차를 계산하고(S260), 그 차가 미리 결정된 문턱값보다 작은 경우 블록은 정지 영역에 위치한다고 판단하고, 움직임 벡터

은 (0, 0)으로 변경한다(S270, S290).Next, the step of correcting the stop motion is performed (S250). Although the value of the actual motion vector is (0, 0), the value of the motion vector may be predicted to a value other than (0, 0). In order to correct this phenomenon, a step of correcting the stationary motion is performed. In this step

and

Calculates the difference of (S260), and if the difference is smaller than the predetermined threshold value, it is determined that the block is located in the still area, the motion vector

Is changed to (0, 0) (S270, S290).

Next, motion prediction is performed again on the enlarged search window (S270 and S280). In the method presented in the present invention, the size of the search window initially used to improve the processing speed is small. If the motion size of the object is larger than the size of the initially used search window, the motion vector is likely to be incorrectly predicted. Thus, it is necessary to search on a larger search window to correct this case. If the SAD value between the block to be processed on the current frame and the block predicted on the previous frame is larger than a predetermined threshold value, the search is performed again on the enlarged search window. In this step, since a search is performed on a region other than the region already searched on the enlarged search window, the execution time is not long.

As mentioned above, the motion prediction method proposed by the present invention is characterized by using a block and a search window of a size suitable for the speed of the target. In addition, since the present invention has been developed for mobile phones, it is necessary to minimize the required hardware complexity. The present invention requires only one frame memory for each color channel while the resulting image according to the present invention has a further improved picture quality than that of the prior art.

Next, a step of interpolating a new frame between a previous frame and a current frame using a motion vector calculated by the motion prediction technique is performed. When the interpolated frame is located in the middle of temporally adjacent frames, a new frame is generated between the previous frame and the current frame by Equation (8). If the pixel value of the interpolated frame is not calculated by Equation 8, the pixel value of the new frame is calculated by Equation 9.

는 최종적으로 계산된 움직임 벡터를 의미하고,

는 생성된 새로운 프레임을 의미한다.here,

Means the finally calculated motion vector,

Means new frame generated.

In order to verify the performance of the embodiment of the present invention, odd-numbered frames are generated from the even-numbered frames of the experimental video sequence according to the embodiment of the present invention. The prior art uses a global search (FS) using a fixed size block and a search region and a three-step search (TSS), which is a fast motion prediction method, and frame interpolation methods are represented by Equations 8 and 9 Is performed by. The size of the blocks used in the experiments of the prior art is 8x8, and the search area is up to 6 pixels in the x and y directions. In order to compare the performance of the prior art and the embodiment of the present invention, PSNR (peak signal to noise ratio) is calculated by Equation 10.

은 원본 홀수 번째 프레임을 나타내고,

은 프레임 레이트 변환 기술에 의하여 생성된 홀수 번째 프레임을 나타낸다. X는 프레임의 전체 화소 개수를 나타낸다.here,

Represents the original odd-numbered frame,

Represents an odd-numbered frame generated by the frame rate conversion technique. X represents the total number of pixels in the frame.

21A to 21C are diagrams showing the PSNRs for a frame generated according to the prior art and the PSNR for a frame generated according to an embodiment of the present invention. The Hall monitor, an experimental video sequence used in FIG. 21A, contains less detail and does not have a large movement of the object. Football, an experimental video sequence used in FIG. 21B, contains a lot of fine areas and has a fast movement of an object. Foreman, the experimental video sequence used in FIG. 21C, has the intermediate characteristics of a Hall monitor and a football. The experimental video sequence consists of 100 frames and each frame is 352 × 288. As shown in FIG. 21, it can be seen that the performance of the result generated according to the embodiment of the present invention is better than the result according to the prior art.

22A and 23A illustrate examples of frames generated by the prior art global search, and FIGS. 22B and 23B illustrate examples of frames generated by the conventional three-step search. 22C and 23C show examples of frames generated according to an embodiment of the present invention. The experimental video sequence of FIGS. 22A-22C is foreman and the experimental video sequence of FIGS. 23A-23C is football.

As described above, the frame rate conversion method according to the present invention has been described with reference to the illustrated drawings, but the present invention is not limited by the embodiments and drawings disclosed herein, and may be applied within the scope of the technical idea. .

1 is a diagram showing an example of converting a low frame rate input video into a high frame rate output video by applying a frame conversion technology of a frame rate conversion technology.

2 is a diagram illustrating an example of a position of a moving object generated by a conventional frame repetition method.

3 is a diagram illustrating an example of a position of a moving object generated by a conventional frame averaging method.

4 is a diagram illustrating an example of an output frame interpolated by a motion compensation average of neighboring frames.

5 is a diagram illustrating an example of a process of predicting a motion vector for a block of size X × Y in a search region of a reference frame.

FIG. 6 is a diagram illustrating 8 × 8 blocks labeled with “a”, “b”, “c”, and “d” in each pixel to explain a conventional subsampling method.

7A and 7B are diagrams illustrating an order of use of four sub-sampling patterns on a search area.

8 is a diagram illustrating a sub-sampled motion field method as a block on a current frame.

9 is a diagram showing that each block is divided into four sub-blocks having a size of N / 2 × N / 2 in the current frame.

10A to 10C are diagrams for describing a process of performing a three step search.

11 is a diagram showing an error surface showing a matching error in a search area,

12A to 12C are diagrams for describing a process of performing a 2D logical rhythm search,

13A and 13B illustrate two search patterns used in diamond search.

14 is a diagram illustrating an example of a process of performing a diamond search.

15 is an overall flowchart for performing a frame rate conversion technique according to the present invention;

16 is an overall flowchart of a motion prediction technique according to the present invention;

17A to 17G are diagrams illustrating each block class and the size of blocks belonging thereto.

18A and 18B illustrate sizes of blocks applied on interpolated frames according to the present invention.

19 is a flowchart illustrating a process of predicting a motion vector of each block using a block matching algorithm.

20A and 20B are diagrams showing examples of subtitles moving at the same speed movement from right to left.

21a to 21c are diagrams showing the PSNR of a frame generated according to the prior art and the PSNR of a frame generated according to an embodiment of the present invention.

22A and 23A illustrate an example of a frame generated according to a prior art global search,

22B and 23B illustrate an example of a frame generated according to a conventional three-stage search.

22C and 23C illustrate examples of frames generated according to an embodiment of the present invention.

Claims (11)

In a frame rate conversion method of converting a low frame rate video into a higher frame rate video by increasing its frame rate, A first step of predicting a motion vector using blocks and search windows of different sizes according to the speed or size of the object; And A second step of converting a low frame rate video into a high frame rate video using the predicted motion vector, The first step may include a first step of predicting a motion activity of a block of a first size at the same position of a previous frame and a current frame; A second process of determining a size of a predetermined block and a search window corresponding to a size and a speed of an object by using the predicted movement activity; And A third process of predicting a motion vector of each block by using a block matching algorithm; The third process may include: a forward motion prediction process of determining a motion vector by finding a location having a minimum matching error with a block of a current frame within a search window of a previous frame; If the difference between the SAD value and the minimum SAD value at each position in the search window of the previous frame is less than the first threshold value, the block of the previous frame within the search window of the current frame. A backward motion prediction process of determining a motion vector by finding a location having a minimum match error with the sigma; A correction process for correcting the static motion; If the minimum matching error between the block to be processed on the current frame and the block predicted on the previous frame is less than the third threshold value, the frame rate conversion method comprising the step of performing the motion prediction again on the extended search window. delete The method according to claim 1, And the first size is 8x8. The method according to claim 1, The method of claim 1, wherein the motion activity is calculated by calculating a summed absolute difference (SAD) for unit blocks located at the same position of the previous frame and the current frame. The method according to claim 4, The second process may include a type determination process of determining whether a block is a static type or a dynamic type according to the SAD value; A block merging process of merging two neighboring identically sized static type blocks to expand a block size; And A frame rate conversion method comprising determining a size of a search window according to a type of a block. The method according to claim 5, The type determining process is determined as a static type when the SAD value is smaller than a predetermined threshold value, and a dynamic type when the SAD value is equal to or larger than the first threshold value. The method according to claim 5, And the second process repeats the block merging process even between merged blocks until a block of a second size. The method of claim 7, And the second size is 32 × 32. The method according to claim 5, The search window size determining process is characterized in that, when the corresponding block is a static type, a search window having a smaller size than that of the dynamic type is used. delete The method according to claim 1, The re-run process is a frame rate conversion method, characterized in that the search is performed for the remaining area except the already searched area.

Images