KR20120004874A - Apparatus and method for upconverting frame rates of decoded video frames - Google Patents

Abstract

PURPOSE: A device and a method for up-converting frame rates of decoded video frames are provided to generate an interpolated frame with high objective/subjective picture quality in a original frame standard. CONSTITUTION: An occlusion section setting unit(120) determines whether a current block which is included in an interpolation frame is a closed interval or not. If the current block is not a closed block, a preliminary frame generator(130) generates the current block from areas. A neighbor information matching unit(140) determines whether the neighboring information of a block inside a closed interval is matched to some part on at least one reference frame. A frame interpolation unit(150) restores a block in the closed interval by the part on the matched reference frame.

Description

Apparatus and method for upconverting frame rates of decoded video frames}

TECHNICAL FIELD The present invention relates to video compression techniques, and more particularly to techniques for interpolating frames in order to upconvert frame rates of reconstructed video.

As information and communication technology including the Internet is developed, not only text and voice but also video communication are increasing. Conventional text-based communication methods are not enough to satisfy various needs of consumers, and accordingly, multimedia services that can accommodate various types of information such as text, video, and music are increasing. The multimedia data has a huge amount and requires a large storage medium and a wide bandwidth in transmission. Therefore, in order to transmit multimedia data including text, video, and audio, it is essential to use a compression coding technique.

The basic principle of compressing data is to eliminate redundancy in the data. Spatial overlap, such as the same color or object repeating in an image, temporal overlap, such as when there is almost no change in adjacent frames in a movie frame, or the same note over and over in audio, or high frequency of human vision and perception Data can be compressed by removing perceptual redundancy, taking into account insensitiveness to. In general video coding techniques, temporal redundancy is eliminated by temporal filtering based on motion compensation, and spatial redundancy is eliminated by spatial transform. The compressed video data is provided to various reconstruction devices (video decoders) through various transmission media. The decompression device obtains a reconstructed video image by performing a process corresponding to an inverse of the compression process on the compressed video data.

However, according to the needs of various reproduction apparatuses or applications, there is a need to change the frame rate of the reconstructed video. If the video frame rate is downconverted, it can be easily implemented by skipping the frame or various other methods.However, if the video frame rate is upconverted, the non-existent frame must be interpolated. The key is how naturally it can be seen by the user. That is, the purpose is to minimize the difference between the frame with the improved frame rate and the frame obtained with the improved frame rate in the original video coding step.

Traditional frame rate up-conversion (FRUC) techniques include frame repetition (FR), frame averaging (FA), and linear frame interpolation (LI). . However, since these techniques do not consider motion vectors, they are low in complexity and easily interpolated. However, when the image format is expanded or the movement of the object is large, ghost artifacts occur in which the image is uneven and the image is overlapped. In addition, since block interpolation is performed, block artifacts may occur at the boundary between blocks.

In order to improve this problem, motion compensation FRUC (MC-FRUC) technology considering motion vectors has been studied. The MC-FRUC is heavily focused on predicting accurate motion vectors, and research has been conducted toward more accurate motion vector prediction. Motion-Compensated Frame Interpolation (MCFI), which is the most basic technology of MC-FRUC, is a method of generating an interpolated frame using a motion vector obtained from a previous frame around a current frame. However, MCFI can efficiently interpolate frames with low motion, but when motion is large, it is difficult to predict accurate motion vectors. In addition, since frame interpolation is performed on a block basis, a block and a boundary portion of the block are not taken into account, and thus a blocking phenomenon appears in the generated image.

In order to solve such a problem, an overlapped block motion compensation (OBMC) has been proposed. The OBMC obtains a smoother effect by varying the motion vector values along the boundary of neighboring blocks to prevent the blockage occurring in MCFI. However, there may be a problem that the image quality is degraded due to excessive overlap of the boundary portion.

The present invention was devised in view of the above-described needs, and an object thereof is to provide a more precise frame interpolation technique for improving a frame rate of reconstructed video.

Technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In the frame rate improving apparatus for improving the frame rate of the reconstructed video frame according to an embodiment of the present invention for achieving the technical problem, based on the motion vector of at least one reference frame of the reconstructed video frame A occlusion section setting unit configured to determine corresponding regions on a plurality of reference frames, and determine whether a current block belonging to an interpolation frame is a occlusion section based on similarities between the corresponding regions; A preliminary frame generation unit generating the preliminary frame by generating the current block from the regions if the current block is not the occlusion section and leaving it empty if the current block is the occlusion section; A peripheral information matching unit configured to determine which part of the at least one reference frame is the peripheral information of the block included in the occlusion section; And a frame interpolation unit for restoring a block included in the occlusion section by a portion on the matched reference frame to generate the interpolation frame.

In the frame rate improving apparatus for improving the frame rate of the reconstructed video frame for achieving the technical problem, using the motion vector of the first reference frame of the reconstructed video frame, and the current block of the frame to be interpolated Occlusion that determines a corresponding first region on the first reference frame and a second region on a second reference frame, and determines whether the current block is an occlusion section based on similarity between the first region and the second region. An interval setting unit; A preliminary frame generator configured to generate a preliminary frame by replacing the current block with the first area when the current block is not the occlusion section; When the current block is the occlusion period, determining which part of the neighboring information of the current block matches on the first reference frame to determine a third region on the first reference frame corresponding to the current block. Peripheral information matching unit; And a frame interpolation unit which interpolates a frame including the current block by replacing the current block with the third region.

In the frame rate improving method for improving the frame rate of the reconstructed video frame to achieve the technical problem, based on a motion vector of at least one reference frame of the reconstructed video frame, corresponding on a plurality of reference frames Determining regions to be; Determining whether a current block belonging to an interpolation frame is an occlusion interval based on the similarity between the corresponding regions; Generating a preliminary frame by generating the current block from the areas if the current block is not a blockage period and leaving the blank if the current block is a blockage period; Determining which part of the at least one reference frame the peripheral information of the block included in the occlusion period matches; And restoring a block included in the occlusion section by a portion on the matched reference frame to generate the interpolated frame.

According to the present invention, an interpolation frame having higher subjective / objective image quality can be generated based on the original frame, compared to various conventional frame interpolation techniques.

1 is a block diagram showing the configuration of an apparatus for improving a frame rate according to an embodiment of the present invention.
2 to 5 are diagrams illustrating a method of determining a corresponding area on a reference frame based on a current block.
6 is a diagram illustrating examples of a first preliminary frame and a second preliminary frame obtained in the case of a foreman sequence.
FIG. 7 is a diagram illustrating a result of merging two preliminary frames illustrated in FIG. 6 by an average.
8 is a diagram illustrating examples of selecting peripheral information of a current block in various cases.
FIG. 9 is a diagram illustrating an example in which peripheral information of a current block (or peripheral information of an area compared on a reference frame) is displayed as a histogram.
FIG. 10 is a diagram illustrating peripheral information about a current block and neighboring information about regions corresponding to the current block on reference frames.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

1 is a block diagram illustrating a configuration of a frame rate improving apparatus 100 for improving a frame rate of a reconstructed video frame according to an embodiment of the present invention.

The frame rate improving apparatus 100 may include a block section setting unit 120, a preliminary frame generation unit 130, a peripheral information matching unit 140, and a frame interpolation unit 150. It may further include. If the frame rate enhancing apparatus 100 is implemented in such a manner that only the frame rate enhancement function is received by receiving the reconstructed video image, the video decoder 110 may not be included.

The video decoder 110 applies conventional video decoding techniques to the input bitstream to recover the original video frame having a predetermined frame rate. The conventional video decoding is a process corresponding to the inverse of video encoding, and may include an inverse entropy encoding step, an inverse quantization step, an inverse transform step, and a prediction decoding (inter prediction / intra prediction decoding) step. Can be understood by those of ordinary skill in the art without any additional explanation. The video frames reconstructed by the video decoder 110 are then processed by the occlusion section setting unit 120, the preliminary frame generation unit 130, the peripheral information matching unit 140, and the frame interpolation unit 150 to improve the frame rate. Go through. The video frame reconstructed by the video decoder 110 may be used as a reference frame (typically, a frame adjacent to the current frame is used) for the current frame to be interpolated (hereinafter, referred to as interpolation frame).

The occlusion section setting unit 120 determines corresponding regions on a plurality of reference frames based on a motion vector of at least one reference frame among the reconstructed video frames, and based on similarities between the corresponding regions, It is determined whether the current block belonging to the interpolation frame is a occlusion section.

According to an embodiment of the present invention, the at least one reference frame means at least one of a frame immediately before or immediately after the interpolated frame (interpolation frame), and the plurality of reference frames are the frame immediately preceding the frame and the frame immediately after the frame. Means. Further, according to a preferred embodiment of the present invention, the reference frame (at least one reference frame and a plurality of reference frames) is two frames immediately before and after the interpolation frame.

The similarity between the corresponding regions may be determined by whether the sum of absolute difference (SAD) of the regions exceeds a predetermined threshold. 2 to 5 are diagrams illustrating a method of determining a corresponding region on a reference frame that is an object of SAD calculation.

FIG. 2 is a diagram illustrating a correspondence relationship between the immediately preceding frame 21 and the immediately following frame 23 using the motion vector 27 of the preceding frame 21 among the reference frames 21 and 23. In this case, the block 25 (current block) to be interpolated currently in the interpolation frame 22, the region 24 of the immediately preceding frame 21 and the region 26 of the immediately following frame 23 are the preceding frame. Corresponds to the motion vector 27 included in the area 24 of (21).

Meanwhile, similar to FIG. 2, FIG. 3 is a diagram illustrating a correspondence relationship between the immediately preceding frame 21 and the immediately following frame 23 using the motion vector 37 of the immediately after frame 23. In this case, the current block 25, the region 34 of the immediately preceding frame 21 and the region 36 of the immediately after frame 23 are the motion vectors of the region 36 of the immediately after frame 23. Corresponding to 37).

However, when obtaining a correspondence relationship in the same manner as in FIGS. 2 and 3, there may not be a motion vector that correctly passes the position of the current block 25. In this case, since an additional complex approximation process is required, undesirable errors may occur.

Accordingly, as shown in FIG. 4, the region 46 on the frame 23 immediately after the motion vector 47 of the region 44 on the immediately preceding frame 21 at the same position as the current block 25 is to be corresponded. Can be. In this manner, the current block 25, the region 44 on the immediately preceding frame 21, and the region 46 on the immediately following frame may simply have a corresponding relationship without a separate approximation process. 4 shows the correspondence using the motion vector of the region 44 on the immediately preceding frame 21, the correspondence using the motion vector 57 of the region 56 on the immediately following frame 23 as shown in FIG. Of course, relationships are possible.

As a result, a corresponding region exists between the previous frame 21 and the immediately following frame 23 by the motion vector of the immediately preceding frame 21 or the motion vector of the immediately following frame 23. The occlusion section setting unit 120 calculates SAD for the corresponding areas, and determines similarity of the corresponding areas by whether or not this exceeds the threshold. That is, if the SAD exceeds the threshold, the error between the corresponding regions is large, so that the reliability of the current block is low. If the SAD does not exceed the threshold, the reliability of the current block is small, the reliability of the current block is high. can see.

Accordingly, the occlusion section setting unit 120 replaces the current block with corresponding areas when the reliability is high, but sets the current block as the occlusion section when the reliability is low, such as the former. The occlusion section means an area where no data has been recorded yet, but in practice, the occlusion section may be assigned with 0 (indicated by a black square on the frame) to distinguish it from other areas.

Here, the threshold may be selected empirically, but it may be more preferable to set the adaptive value suitable for various images. According to an embodiment of the present invention, the threshold value may be obtained from covariance of pixel values of the corresponding regions. That is, a value obtained by multiplying the covariance by a proportional constant may be used as a threshold.

Meanwhile, in FIG. 1, the preliminary frame generation unit 130 generates the current block from the regions if the current block is not a blockage period and reserves (or fills with zeros) if the current block is a blockage period. Create a frame. As described above, one preliminary frame (first preliminary frame) may be generated using the motion vector of the immediately preceding frame, and another preliminary frame (second preliminary frame) may be generated even using the motion vector of the immediately following frame. Since this can be generated, there may be two spare frames.

FIG. 6 is a diagram illustrating examples of a first preliminary frame (FIG. 6A) and a second preliminary frame (FIG. 6B) obtained in the case of a foreman sequence. Each preliminary frame is marked with its own occlusion section (indicated by a black square). For each preliminary frame, the occlusion sections show a somewhat similar pattern but do not completely match. As described above, when the generated preliminary frames are plural, the preliminary frame generator 130 may merge the plurality of preliminary frames by averaging the plurality of preliminary frames. Since the occlusion section has a value of 0, the merging by the mean means that when all pixels corresponding to the two preliminary frames belong to the occlusion section, the pixels in the merged frame also become the occlusion section, and only one of the two pixels is included. With normal pixels, the pixels in the merged frame are replaced with the normal pixels. If both pixels have normal pixels, it means that the pixels in the merged frame are calculated as the average of the normal pixels. The result of merging two preliminary frames (a) and (b) shown in FIG. 6 by an average (merged frame) is shown in FIG. 7.

Referring back to FIG. 1, the neighbor information matching unit 140 determines which part of the at least one reference frame the peripheral information of the block included in the occlusion section (the current block included in the occlusion section) matches. . In particular, the surrounding information matching unit 140 may use the sum of absolute histogram differences rather than using SAD again to determine whether the matching is performed. However, this does not exclude SAD and other matching methods.

The peripheral information is pixel values having a predetermined reference range adjacent to the periphery of the current block (which belongs to the occlusion section 25) included in the occlusion section, and shades 81 to 85 in various cases illustrated in FIG. 8. Is displayed.

8A to 8E show cases where the number of blocks belonging to the occlusion section in the vicinity of the current block 25 is 0 to 4, respectively. In one embodiment of the present invention, the reference range becomes larger as the occlusion section increases around the block included in the occlusion section. As the line of pixels further away from the line of the pixels closest to the current block is added, the reference range is increased. For example, in the case of FIGS. 8A to 8E, the reference ranges are 2 to 6, respectively. That is, as the number of neighboring blocks belonging to the occlusion section increases, more reference ranges are used. If a constant reference range is used regardless of the number of neighboring blocks belonging to the occlusion section, as the number of neighboring blocks belonging to the occlusion section increases, the surrounding information to be referred to decreases, thereby reducing the overall reliability. In the matching process, such peripheral information is compared in the interpolation frame and the same shape and number in the reference frame.

On the other hand, in order to obtain the sum of the above absolute histogram difference, it is first necessary to generate the histogram as shown in FIG. For example, the peripheral information of the current block (peripheral information as illustrated in FIG. 8) is a set of a plurality of pixel values, which are divided into N bins to indicate respective frequencies and similarly referred to as a reference. The peripheral information of the region on the frame is similarly displayed in this manner. The sum of the absolute histogram differences is obtained by calculating the absolute value of the difference between the bins of the neighboring information of the current block and the bins of the neighboring information on the compared reference frame (Bin _j (j = 1, 2, ..., N) difference), means the result of the sum of these absolute values.

The peripheral information matching unit 140 determines that the current block matches the region on the reference frame when the sum of the absolute histogram differences becomes the minimum. Of course, the range for searching for such a match will be a range of a predetermined size centering on the region determined to correspond in the above-described occlusion section determination process. Through such a matching process, as illustrated in FIG. 10, regions corresponding to reference frames 21 and 23 corresponding to the current block 25 on the interpolation frame 22 (previous and immediately after reference frames) 21 and 23 ( 91, 93) is determined. Here, the shape of the peripheral information 81 for the current block 25 is the same as that of the peripheral information 92 and 94 for the corresponding areas 91 and 93.

Referring back to FIG. 1, the frame interpolator 150 restores a block included in the occlusion section by a portion on the matched reference frame to generate the interpolation frame. However, when the above-described preliminary frames are plural, the frame interpolator 150 may also have two parts on the matched reference frame, so that the frame interpolator 150 may include a part on the matched first reference frame, The occlusion section may be finally restored by averaging portions on the matching second reference frame. As a result, when all the occlusion sections are restored on the existing merged preliminary frame, a final interpolation frame can be generated.

Until now, each component of FIG. 1 may refer to software, or hardware such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the components are not limited to software or hardware, and may be configured to be in an addressable storage medium and configured to execute one or more processors. The functions provided in the above components may be implemented by more detailed components, or may be implemented as one component that performs a specific function by combining a plurality of components.

In order to verify the effect according to the present invention, the performance and performance of conventionally known technologies such as LI, MCFI, OBMC and Robust-FRUC were compared and analyzed, and the results are summarized in Table 1 below. The experimental images were all CIF (352x288) size, and the predicted block size was fixed at 8x8, and the search area in the step for judging the occlusion section was 16x16 and was searched up to 1/4 pixel. The search region in the step of predicting the motion vector using the information of the neighboring block is set to 2 × 2 so that the median value of the motion vector of the neighboring block can be considered.

Video sequences PSNR LI MCFI OBMC Robust-FRUC Invention Foreman 29.3092 32.7338 33.1404 33.4895 34.4829 Bus 18.9636 21.6715 21.9122 22.6575 22.8798 Flower 19.8022 31.2147 31.2147 31.8080 32.6242 Mobile 24.6965 29.2996 29.2996 30.8095 31.8043 Children 26.2263 28.9145 28.9650 29.6032 29.8559 City 24.0753 29.5009 29.8009 30.1538 30.7650 Container 41.8994 43.3318 43.3807 43.4022 43.6108 Football 20.5605 22.6757 22.8751 23.4082 23.9764 Monitor 36.3804 36.8180 36.9649 37.3367 37.6899 Soccer 19.5316 20.2045 20.5903 20.9306 21.6223 Average 27.8689 31.2220 31.4616 31.9043 32.2685

Table 1 shows that the Peak Signal-to-Noise Ratio (PSNR) according to the present invention is improved in performance over existing technologies. LI has significantly lower PSNR performance than other technologies. The OBMC is a method for improving the blockage phenomenon of the existing MCFI, but shows an improved result in terms of PSNR, but as described above, image degradation occurs due to excessive duplication of blocks. Robust-FRUC also slightly improves the performance of PSNR, but cannot achieve significantly improved results compared to the increase in the amount of computation using bidirectional motion information.

In contrast, the technique according to the invention shows significantly improved results compared to existing techniques. According to the present invention, Foreman and Mobile video are 1.4dB and 2.5dB higher than MCFI and 0.99dB higher than Robust-FRUC. In addition, even compared to Robust-FRUC, it can be seen that there was an improvement of image quality by 0.6dB on average.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

100: frame rate enhancement device 110: video decoder
120: block section setting unit 130: preliminary frame generation unit
140: peripheral information matching unit 150: frame interpolation unit

Claims (16)

In the frame rate improving apparatus for improving the frame rate of the reconstructed video frame,
Based on a motion vector of at least one reference frame of the reconstructed video frame, corresponding regions are determined on a plurality of reference frames, and based on similarity between the corresponding regions, the current block belonging to the interpolated frame is occluded. An occlusion section setting unit determining whether the section is a section;
A preliminary frame generation unit generating the preliminary frame by generating the current block from the regions if the current block is not the occlusion section and leaving it empty if the current block is the occlusion section;
A peripheral information matching unit configured to determine which part of the at least one reference frame is the peripheral information of the block included in the occlusion section; And
And a frame interpolation unit for restoring a block included in the occlusion section by a portion on the matched reference frame to generate the interpolation frame. The method of claim 1,
The at least one reference frame means at least one of a frame immediately before or immediately after the interpolated frame, and the plurality of reference frames mean the frame immediately before the frame and the frame immediately after the frame. The method of claim 1, wherein the similarity
And a sum of absolute difference between the corresponding regions is determined by a predetermined threshold value. The method of claim 3, wherein the threshold is
And a frame rate improving device obtained by covariance of pixel values of the corresponding regions. The method of claim 1, wherein the preliminary frame generation unit
And when the generated preliminary frames are plural, merging the plurality of preliminary frames by averaging them. The method of claim 1, wherein the peripheral information matching unit
The frame rate improving apparatus using the sum of the absolute histogram difference in determining the matching. The method of claim 1, wherein the surrounding information is
And a pixel value having a predetermined reference range adjacent to a block included in the occlusion section. The method of claim 7, wherein the reference range is
The frame rate improving device is larger as the occlusion section increases around the block included in the occlusion section. The method of claim 1, wherein the frame interpolation unit
And reconstructing the block by an average of portions on the matched reference frame when the generated preliminary frames are plural. In the frame rate improving apparatus for improving the frame rate of the reconstructed video frame,
The first region on the first reference frame and the second region on the second reference frame corresponding to the current block of the frame to be interpolated are determined by using the motion vector of the reconstructed video frame. An occlusion section setting unit that determines whether the current block is an occlusion section based on a similarity between a first region and the second region;
A preliminary frame generator configured to generate a preliminary frame by replacing the current block with the first area when the current block is not the occlusion section;
When the current block is the occlusion period, determining which part of the neighboring information of the current block matches on the first reference frame to determine a third region on the first reference frame corresponding to the current block. Peripheral information matching unit; And
And a frame interpolation unit which interpolates a frame including the current block by replacing the current block with the third region. In the frame rate improving method for improving the frame rate of the reconstructed video frame,
Determining corresponding regions on a plurality of reference frames based on a motion vector of at least one reference frame of the reconstructed video frame;
Determining whether a current block belonging to an interpolation frame is an occlusion interval based on the similarity between the corresponding regions;
Generating a preliminary frame by generating the current block from the areas if the current block is not a blockage period and leaving the blank if the current block is a blockage period;
Determining which part of the at least one reference frame the peripheral information of the block included in the occlusion period matches; And
Restoring a block included in the occlusion section by a portion on the matched reference frame to generate the interpolated frame. The method of claim 11, wherein the similarity is
And a sum of absolute difference between the corresponding regions is determined by a predetermined threshold value. The method of claim 12, wherein the threshold is
And a frame rate improving method obtained by covariance of pixel values of the corresponding regions. 12. The method of claim 11, wherein determining whether the match is
And calculating a sum of absolute histogram difference. The method of claim 11, wherein the surrounding information is
And a pixel value having a predetermined reference range adjacent to a block included in the occlusion section. The method of claim 15, wherein the reference range is
The frame rate improving method is larger as the occlusion section increases around the block included in the occlusion section.

Images