KR20140134907A - Apparatus and method for Motion Compensated Interpolation of moving caption region - Google Patents

Abstract

According to an embodiment of the present invention, a method for interpolating a frame using motion compensation of a moving caption area includes the steps of: extracting primary caption candidate area image information from inputted consecutive image sequences using temporal features and spatial features; generating an edge image based on the primary caption candidate area image information and extracting a secondary caption candidate area based on the edge image; and verifying a caption area by comparing the secondary caption candidate area and the image sequences.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion compensated frame interpolation apparatus and a motion compensated frame interpolation method,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a frame interpolation apparatus and method, and more particularly, to a frame interpolation apparatus and method for a display quality enhancement in image processing of frame rate conversion .

Frame Rate Conversion is one of the most important issues in recent years due to the explosive increase in image information and display formats. Frame rate conversion techniques are needed for conversion between various display formats with different frame rates. For example, existing videos have frame rates of 24, 25, and 30 per second, but HDTV and multimedia PC systems support higher frame rates to reduce image quality degradation such as screen blinking and improve output quality.

Therefore, when the video is played back in the HDTV or multimedia environment, the higher the frame rate, the better the picture quality can be expected. Particularly, in the case of an LCD display device, deterioration of image quality may be prevented by increasing the frame rate to be displayed even though a deterioration of the moving image quality due to a slow response speed occurs.

In the past, simple algorithms such as frame repetition or linear interpolation with temporal filtering were used for frame up-conversion. However, these simple algorithms cause image degradation such as motion judder and motion blur. In order to prevent such deterioration of image quality, various motion compensation frame rate conversion algorithms using block-based motion compensation interpolation techniques have been developed. In such a motion compensation scheme, it is very important to estimate the motion vectors of the objects in the image because all the interpolation processes are performed by the motion vectors.

However, in block-based motion estimation, it is often impossible to estimate an accurate motion vector for reasons such as noise, brightness change, presence of a local minimum value due to the presence of a similar image, object shading, change of object shape, and the like.

In particular, in the case of an image having a moving subtitle, moving subtitles and objects moving in different directions are present in one block at the same time, so that accurate motion estimation is difficult. For this reason, incorrectly estimated motion vectors cause image degradation such as block artifacts in the interpolated image.

If the frequency is extremely low, image degradation due to erroneous motion vector estimation can be ignored due to various merits of the motion compensation interpolation technique. However, in the conventional block-based motion compensation frame rate conversion technique, Occurs very frequently and causes a large amount of block artifacts, which can seriously disturb the viewer's eyes. Especially, when a block artifact occurs in the subtitle area, it is easier for viewers to recognize than other scenes. In this case, the decrease in PSNR due to the artifacts of the misinterpreted subtitles is not so large, but it takes a very large portion in the subjective image quality evaluation, and a separate processing technique is required.

As described above, in order to prevent block artifacts due to failure of motion estimation due to failure of accurate motion estimation when a moving subtitle and a moving object exist in one block at the time of motion estimation of a moving subtitle area, And a method of interpolating motion compensation of a moving subtitle area irrespective of the motion of an object.

It is to be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may be present.

According to an aspect of the present invention, there is provided an apparatus for interpolating a motion compensated frame, comprising: a first subtitle candidate region image A moving subtitle candidate region extracting unit for extracting a moving subtitle candidate region; A moving subtitle feature detector for generating a secondary subtitle candidate area using edge information according to a contour characteristic of a character from the extracted primary candidate region image; And a mobile caption region verification unit for verifying a secondary caption candidate region generated by the mobile caption feature detection unit from the input image.

According to another aspect of the present invention, there is provided a method for interpolating motion compensation frames in a moving subtitle area, the method comprising the steps of: Extracting first caption candidate region image information using the first caption candidate region image information; Generating an edge image based on the first subtitle candidate region image information and extracting a second subtitle candidate region based on the edge image; And verifying the caption region by comparing the secondary caption candidate region and the video sequence.

According to the embodiment of the present invention, it is possible to prevent a motion artifact estimated from a motion estimation apparatus for a block-based motion compensation frame rate conversion scheme from erroneously estimating a blocking artifact due to the simultaneous presence of a moving subtitle and a moving object within one block We have proposed a technique to detect the subtitle area and apply it to a different vector from the surrounding area in the future interpolation.

Accordingly, when the moving subtitle and the moving object exist at the same time, the blocking artifact caused by the incorrect estimation of the motion vector can be minimized. In particular, when block artifacts occur in the caption region, the block artifact minimization effect of the static caption region through frame correction can be maximized because the viewer is more perceptible than other cases.

1 is a flowchart illustrating a method of detecting a moving subtitle area in an image including a moving subtitle according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining a moving subtitle candidate region extracting unit according to the present invention in detail.
FIG. 3 is a diagram for explaining a moving subtitle feature extracting unit according to the present invention in detail.
4A to 4I are views for explaining a moving subtitle candidate region extracting unit according to an embodiment of the present invention in detail.
5A to 5E are diagrams for explaining a moving subtitle feature extracting unit according to an embodiment of the present invention in detail.
FIG. 6A is an example of a frame interpolation image to which the present invention is not applied, and FIG. 6B is an example of a frame interpolation image to which the present invention is applied.
7 is a view for explaining a motion compensation frame apparatus according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Thus, those skilled in the art will be able to devise various apparatuses which, although not explicitly described or shown herein, embody the principles of the invention and are included in the concept and scope of the invention. Furthermore, all of the conditional terms and embodiments listed herein are, in principle, intended only for the purpose of enabling understanding of the concepts of the present invention, and are not intended to be limiting in any way to the specifically listed embodiments and conditions .

It is also to be understood that the detailed description, as well as the principles, aspects and embodiments of the invention, as well as specific embodiments thereof, are intended to cover structural and functional equivalents thereof. It is also to be understood that such equivalents include all elements contemplated to perform the same function irrespective of the currently known equivalents as well as the equivalents to be developed in the future, i.e., the structure.

Thus, for example, it should be understood that the block diagrams herein represent conceptual views of exemplary circuits embodying the principles of the invention. Similarly, all flowcharts, state transition diagrams, pseudo code, and the like are representative of various processes that may be substantially represented on a computer-readable medium and executed by a computer or processor, whether or not the computer or processor is explicitly shown .

The functions of the various elements shown in the figures, including the functional blocks depicted in the processor or similar concept, may be provided by use of dedicated hardware as well as hardware capable of executing software in connection with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.

Also, the explicit use of terms such as processor, control, or similar concepts should not be interpreted exclusively as hardware capable of running software, and may be used without limitation as a digital signal processor (DSP) (ROM), random access memory (RAM), and non-volatile memory. Other hardware may also be included.

In the claims hereof, the elements represented as means for performing the functions described in the detailed description include all types of software including, for example, a combination of circuit elements performing the function or firmware / microcode etc. , And is coupled with appropriate circuitry to execute the software to perform the function. It is to be understood that the invention defined by the appended claims is not to be construed as encompassing any means capable of providing such functionality, as the functions provided by the various listed means are combined and combined with the manner in which the claims require .

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, in which: There will be. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a motion compensation interpolator according to an embodiment of the present invention.

Referring to FIG. 1, a motion compensation interpolator according to the present invention includes a subtitle candidate region extraction unit 100, a subtitle feature extraction unit 200, and a subtitle region verification unit 300.

First, the moving subtitle candidate region extracting unit 100 receives a video sequence and generates a first moving subtitle candidate region image from the input video sequence. The moving subtitle candidate region extracting unit 100 may analyze the inputted video sequence according to the temporal and spatial characteristics of the moving subtitle and generate the first moving subtitle candidate region image according to the analysis result.

The moving picture candidate region extracting unit 100 includes a video brightness difference calculating unit 120, a video brightness difference minimum calculating unit 140, a horizontal caption region detecting unit 160, a vertical caption region detecting unit 180, . ≪ / RTI >

The moving subtitle feature extracting unit 200 extracts a precise secondary moving subtitle from the primary moving subtitle candidate region using the contour characteristic of the character in the primary moving subtitle candidate region image generated by the moving subtitle candidate region extracting unit 100, Create a candidate region.

Then, the moving subtitle region verification unit 300 uses the input video sequence to extract the moving subtitle candidate region in the reference frame using the secondary moving subtitle candidate region generated by the moving subtitle feature extraction unit 200 as the moving speed of the subtitle And then compared with the reference frame to verify that the caption area is the correct area, and the verified area is determined as the final caption area and output.

When the final moving subtitle area is determined, the motion compensation interpolator can correct the motion vector for the last moving subtitle area in a specific direction and speed corresponding to the moving subtitle in the interpolation process through motion compensation. For example, the frame interpolator can interpolate the corrected block with an average value with respect to the previous frame. As a result, unlike other objects, the moving artifact region is corrected to a specific motion vector, thereby minimizing block artifacts in the moving subtitle region.

FIG. 2 is a block diagram for explaining the moving subtitle candidate region extracting unit 100 in more detail, and FIG. 3 is a block diagram for explaining the moving subtitle characteristic extracting unit 200 in more detail.

2, the moving subtitle candidate region extracting unit 100 includes a video brightness difference calculating unit 120, a video brightness difference minimum calculating unit 140, a horizontal subtitle region detecting unit 160, And an area detection unit 180.

The image brightness difference value calculator 120 receives the next frame of the reference frame as a reference frame and the reference frame as an input image, moves the reference frame in units of pixels, and generates an absolute value of the difference between the two images.

The minimum image brightness calculation unit 140 generates a distance between the pixels having the smallest difference by calculating how many pixels of the reference frame are shifted for each pixel to determine whether the absolute value of the difference between the pixel brightness of the two images is the smallest After subtracting only the distance information suitable for the moving speed of subtitles on average, it removes the excessively small areas to be viewed as subtitles.

Figures 4A-4I show images processed in accordance with one embodiment of the present invention in relation to the minimum image brightness.

4A and 4B show an example of an input image sequence. FIG. 4C is a diagram illustrating distance information between pixels having a minimum difference generated by the minimum image brightness calculating unit 140 according to a difference value output from the image brightness difference calculating unit 120. FIG. As shown in FIG. 4C, it can be seen that the difference between the pixels has a similar value for the caption area

4D is a view showing only the distance information suitable for the moving speed of the caption, and FIG. 4E shows the filtered image by removing regions smaller than the preset caption region size in FIG. 4D. And may be generated in the minimum image brightness calculating unit 140.

Referring back to FIG. 2, the horizontal caption region detection unit 160 detects only information in a horizontal direction using a distance information suitable for the caption generated by the video brightness minimum value calculation unit 140.

The vertical caption region detection unit 180 detects only information in a vertical direction from the horizontal direction distance information generated by the horizontal caption region detection unit 160.

FIG. 4F is a diagram illustrating the number of horizontal direction frequencies of the effective inter-vehicle distance information generated as shown in FIG. 4E. FIG. 4G is a diagram showing only a region generated in the horizontal direction caption region detection unit 160 and having a frequency of more than a predetermined ratio in the horizontal direction in FIG. 4E.

4H is a diagram showing the frequency of valid distance information in the vertical direction in FIG. 4G as a result generated by the vertical caption region detecting unit 180. FIG.

4I is a picture showing a rectangular area using information generated by the vertical direction caption region detection unit 180 in the input reference frame, and is a primary moving caption candidate region image generated by the movement caption candidate region extraction unit 100 .

As shown in FIGS. 4F to 4I, the motion compensation interpolator according to the embodiment of the present invention performs projection in the horizontal direction using an area suitable for the moving speed of the caption, discards a region having a frequency less than a predetermined frequency, Can be obtained. Then, by using projection results in the horizontal direction, projection can be calculated in the vertical direction in the same manner to find a valid caption candidate region. Generally, since the caption area is a rectangular area, the smallest rectangular area including the result of projecting in the vertical direction can be a moving subtitle candidate area. In the embodiment of the present invention, since the pixel minimum value may not be accurately detected at the boundary portion of the caption, the first candidate region may be determined by enlarging the rectangular region at a predetermined ratio.

As described above, the secondary moving subtitle candidate region is generated more precisely than the primary moving subtitle candidate region using the contour characteristics of characters in the primary moving subtitle candidate region image generated by the moving subtitle candidate region extracting unit 100 .

3, the moving subtitle feature extraction unit 200 may include a subtitle contour detection unit 220, a horizontal subtitle contour area detection unit 240, and a vertical subtitle contour area detection unit 260 .

The subtitle contour detecting unit 220 generates a contour information image in consideration of the contour characteristics of characters in the primary moving subtitle candidate region image generated by the moving subtitle candidate region extracting unit 100. [

FIG. 5A is a view showing an image generated after the moving picture candidate region extracting unit 100 of FIG. 4A is performed, and FIG. 5B is an edge picture generated after performing the caption contour detecting unit 220 of FIG. 5A.

The horizontal subtitle contour area detecting unit 240 uses the contour information generated after the subtitle contour detecting unit 220 to calculate a line whose contour line number is greater than a predetermined ratio for the first time as a first line, Only the information of the area to be the last line is detected.

The vertical-direction subtitle contour area detecting unit 260 detects, as the first line, a line in which the number of contours in the vertical direction is greater than a predetermined ratio for the first time among the contour information generated in the horizontal-direction subtitle contour area detecting unit 240, As the last line.

FIG. 5C is a diagram showing only lines having a frequency greater than a predetermined ratio in the horizontal direction among the outline information of FIG. 5B.

FIG. 5D is a diagram showing only lines having a frequency greater than a predetermined ratio in the horizontal direction among the horizontal direction contour information of FIG. 5C.

FIG. 5E is a diagram showing a rectangular area including information generated by the vertical subtitle contour area detecting unit 260, and shows a secondary moving subtitle candidate area generated by the moving subtitle feature extracting unit 200. FIG.

As shown in FIGS. 5A to 5E, the moving subtitle feature extracting unit 200 can generate an edge map to find the boundary portion of the correct subtitles from the first subtitle candidate region. After calculating edge distributions in the horizontal and vertical directions described above from the edge map, it is possible to find a line where an edge at a certain rate or more is first detected in the vertical and horizontal directions, and the area including these lines is referred to as a second subtitle candidate area .

As described above, the moving subtitle region verifying unit 300 uses the input video sequence to search for the moving subtitle candidate region in the reference frame using the secondary moving subtitle candidate region generated by the moving subtitle characteristic extracting unit 200 It is possible to verify whether the corresponding caption area is a correct area and to determine the verified area as a final moving caption area by comparing the moving speed of the caption with the reference frame. The detected caption area is applied to a different vector from the neighboring blocks in the motion compensation interpolation, so that the frame can be used for the interpolation.

6A is a frame interpolation result image when the frame interpolation technique of the present invention is not applied, and FIG. 6B is a frame interpolation result image when the frame interpolation technique of the present invention is applied. 6A and 6B, it can be seen that the image quality of the interpolated image is improved when the motion compensation frame interpolation scheme of the moving subtitle area according to the present invention is used.

As described above, in order to prevent the motion vector estimated from the motion estimation apparatus for the block-based motion compensation frame rate conversion scheme from being erroneously estimated due to the simultaneous presence of the moving subtitles and moving objects within one block, blocking artifacts are generated, It is possible to propose a technique of applying a vector different from the surrounding vector in the subsequent interpolation.

Frame rate conversion techniques are emerging as an important issue due to the explosive increase of video information and display formats, but simple algorithms such as past frame repetition and linear interpolation through temporal filtering cause image degradation such as motion judder and motion blur .

Therefore, even if the motion vector of the block including the current subtitle is wrongly estimated, the motion vector of the block detected as the subtitle area is set again The block artifacts of the moving subtitle area can be minimized.

7 shows a configuration of a frame interpolator according to an embodiment of the present invention.

As shown in FIG. 7, the frame interpolator may include a motion caption region detection module, a motion prediction module, a motion vector correction module, and a motion compensation unit.

As described above, the mobile caption area detection module receives two sequential input images, calculates the distance information between two pixels, finds an area suitable for the moving speed of the statistical caption, And the first candidate region can be generated as an area having a predetermined size. After converting the first candidate region into an edge image, a second candidate region is generated in which an edge having a certain ratio or more in the horizontal and vertical directions is detected. If the second candidate region is compared with the actual input image and a difference of less than a predetermined value is detected, it is detected as an actual moving subtitle, and information on the detected moving subtitle region can be output to the motion vector correction module.

The motion prediction module predicts a motion vector for a frame to be interpolated based on inter-frame motion information to perform interpolation according to the frame rate conversion.

The motion vector correction module receives the moving subtitle area information from the still caption area detection module and corrects the motion vector of the block included in the moving subtitle of the motion vector predicted by the motion prediction module to a specific value. For example, the motion vector correction module may correct the motion vector value of the moving subtitle area to the mode of the motion vector of the corresponding area.

The motion compensation unit performs inter-frame interpolation through motion compensation based on a motion vector output from the motion vector correction module. In particular, the motion compensation unit can interpolate the corrected block using the corrected motion vector.

In this manner, the block artifacts of the moving subtitle area can be minimized by first detecting the moving subtitle area in the video sequence and correcting the moving subtitle area with a specific vector using the consistency of the moving distance information of the moving subtitle area.

The method according to the present invention may be implemented as a program for execution on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include a ROM, a RAM, a CD- , A floppy disk, an optical data storage device, and the like, and may also be implemented in the form of a carrier wave (for example, transmission over the Internet).

The computer readable recording medium may be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner. And, functional programs, codes and code segments for implementing the above method can be easily inferred by programmers of the technical field to which the present invention belongs.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It should be understood that various modifications may be made by those skilled in the art without departing from the spirit and scope of the present invention.

100: moving subtitle candidate region extracting unit 120: image brightness difference value calculating unit
140: image brightness difference minimum value calculation unit 160: horizontal direction caption area detection unit
180: vertical direction caption region detection unit 200:
220: caption contour detection unit 240: horizontal caption contour area detection unit
260: longitudinal subtitle contour area detecting unit 300: moving subtitle area verifying unit

Claims (13)

A motion compensated frame interpolator,
A moving subtitle candidate region extracting unit for extracting a first subtitle candidate region image according to a temporal feature and a positional feature from a continuously input image;
A moving subtitle feature detector for generating a secondary subtitle candidate area using edge information according to a contour characteristic of a character from the extracted primary candidate region image; And
And a mobile caption region verifying unit for verifying a second caption candidate region generated by the mobile caption feature detecting unit from the input image. The apparatus of claim 1, wherein the moving subtitle candidate region extracting unit
Wherein the first subtitle candidate region image is generated based on the distance information by comparing two images among the input images to obtain distance information between pixels having the smallest difference value, Device. The apparatus of claim 1, wherein the moving subtitle candidate region extracting unit
An image brightness difference value calculation unit for moving two images of the input image in units of pixels and outputting a difference value;
Calculating a moving distance between pixels at a position corresponding to a minimum difference value among the brightness difference values output from the image brightness difference value calculation unit and calculating a minimum brightness difference value for outputting position information according to a distance related to the moving subtitle part;
A horizontal caption region detecting unit for detecting region information in which an edge in a horizontal direction is not less than a predetermined rate from position information output from the image brightness difference minimum value calculating unit; And
And a vertical caption region detecting unit for detecting, from the region information detected from the horizontal caption region detecting unit, region information in which the edge in the vertical direction is not less than a predetermined ratio. 4. The apparatus of claim 3, wherein the image brightness difference value calculation unit
An absolute value of a difference between the reference frame and the reference frame is generated by calculating a difference value by moving the reference frame in units of pixels using a next frame of the reference frame included in the input image as a reference frame, Wherein the motion compensating frame interpolator comprises: The apparatus of claim 3, wherein the image brightness difference minimum value calculation unit
The moving distance of the pixel having the smallest difference value among the difference values output from the image brightness difference calculating unit is extracted and only the area information having the moving distance corresponding to the moving speed of the subtitle is extracted, Wherein the motion compensating frame interpolating unit removes the motion compensating frame interpolating unit. The apparatus of claim 3, wherein the transverse caption area detecting unit
And generates horizontal direction area information in which an edge in a horizontal direction is detected at a predetermined ratio or more from the moving distance information of the pixel output from the image brightness difference minimum value calculation unit. The apparatus of claim 3, wherein the vertical caption region detecting unit
A vertical direction region information in which an edge in the vertical direction is detected at a predetermined ratio or more from the horizontal direction region information outputted from the horizontal direction caption region detection unit is extracted and a rectangular region corresponding to the detected horizontal direction region information and vertical direction region information is generated And generates area information on the rectangular area from the reference frame of the input image. The method according to claim 1, wherein the moving subtitle characteristic extracting unit
A subtitle contour detection unit for generating edge video information from the first subtitle candidate area information output from the moving subtitle candidate area extraction unit;
A horizontal subtitle contour detection unit for outputting horizontal direction region information based on the edge video information output from the subtitle contour detection unit; And
And a vertical subtitle contour detection unit for extracting an area in which the edge in the vertical direction is not less than a predetermined ratio from the horizontal direction area information output from the horizontal subtitle contour detection unit to generate a secondary subtitle candidate area. A motion compensated frame interpolator. The apparatus of claim 8, wherein the subtitle contour detection unit
And edge image information is generated using the character contour characteristics from the primary subtitle film candidate area information output from the moving subtitle film candidate area extraction unit. The apparatus of claim 8, wherein the horizontal subtitle contour area detecting unit
And outputs the information of the line where the edge image information is firstly detected and the information of the line which is finally detected from the edge image information outputted from the annoying subtitle contour detection section as the horizontal direction area information. Frame interpolator. 9. The apparatus according to claim 8, wherein the vertical subtitle contour area detecting unit
A rectangular-shaped shape calculated by using information of a line in which edge image information is first detected at a predetermined ratio or more in the vertical direction and information of the last detected line based on the horizontal direction area information output from the horizontal-direction subtitle contour area detecting unit Region to the second subtitle candidate area. The apparatus of claim 1, wherein the moving subtitle area verifying unit
Wherein the second subtitle candidate region information output from the moving subtitle feature extraction unit verifies whether the subtitle region detected from the video sequence is the correct subtitle region and outputs the final subtitle region. A motion compensation frame interpolation method for a moving subtitle area,
Extracting first subtitle candidate region image information using a temporal feature and a positional feature in a continuous video sequence input as an input;
Generating an edge image based on the first subtitle candidate region image information and extracting a second subtitle candidate region based on the edge image; And
And comparing the second subtitle candidate area with the video sequence to verify a subtitle area.

Images