KR100856215B1 - Method and apparatus for frame error concealment in video decoding - Google Patents

Abstract

The present invention provides a method for concealing frame loss during video decoding, comprising: checking a loss of an input frame during video decoding, and reconstructing the input frame in macroblock units when a loss occurs in the input frame. Setting the macro block at the same position as the macro block to be restored in the frame and surrounding macro blocks up, down, left, and right of the macro block at the same position as the reference macro block to be used for restoration, and examining each motion vector of the reference macro blocks. Applying a predetermined weight to the examined motion vector, calculating a weighted average motion vector, restoring the lost macroblock using the weighted average motion vector, and restoring all the macroblocks of the lost frame Includes the process of creating a reconstructed frame to replace the frame. The.

Frame, loss concealment, motion vector, macro block

Description

Method and apparatus for concealing frame loss in video decoding {METHOD AND APPARATUS FOR FRAME ERROR CONCEALMENT IN VIDEO DECODING}

1 is a diagram illustrating frames output after a frame error occurs during video decoding.

2 is an exemplary diagram of frames output after a frame error occurs when decoding a video using a general Frame Freeze technique.

3 is a block diagram illustrating a frame loss concealment apparatus when decoding a video according to an embodiment of the present invention.

4 is an exemplary view illustrating a method of referring to a macroblock of a previous frame of a frame loss concealment method when decoding a video according to an embodiment of the present invention.

FIG. 5 is a diagram illustrating weights of motion vectors of blocks in a method for concealing frame loss during video decoding according to an embodiment of the present invention; FIG.

6 is a flowchart of a frame loss concealment operation when decoding a video according to an embodiment of the present invention.

7 is a diagram illustrating frames output after a frame error occurs when decoding a video using the frame loss concealment method according to an embodiment of the present invention.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to video decoding, and more particularly, to a method and apparatus for concealing a loss of a frame (EC) during video decoding.

In general, most video coding techniques, such as H.263, H.264, and MPEG-4, which have been established as international standards, are effective in variable length coding (VLC) and spatial and motion prediction. I am using image compression technique. However, the variable length coding technique is very vulnerable to a lossy channel environment, which causes the loss of synchronization during image decoding even by a few bit losses, and the predictive coding technique causes the loss to propagate through the entire image. As a result, the loss incurred during channel transmission, storage medium recording, or reading has a great effect on the deterioration of the restored image.

In order to effectively cope with such unavoidable transmission loss, loss concealment (EC) is studied as a technique to minimize the damage caused by the loss in the transmission environment. Has been.

The most widely used technique is the post-processing technique in which the decoder plays the role of EC. In the case of a natural image, spatially or temporally adjacent pixels are smoothly restored to recover lost information. . At this time, the lost block is reconstructed in pixel units or macro block units by using pixel values or motion vectors (MVs) of neighboring blocks. However, in the case of an error generated in units of frames, unlike an error in units of blocks, there is a problem in that no information is available in one frame. For this reason, there are almost no EC techniques in frame units.

1 is a diagram illustrating frames output after a frame error occurs during video decoding. Referring to FIG. 1, a lost frame F _k and a frame transmitted thereafter show a form in which an error of a previous frame is propagated due to a compression scheme using correlation with a previous frame. Therefore, the lost frame is F _k, but the effect is transmitted to the later transmitted F _{k + 1} and F _{k + 2} frames, making it impossible to check the actual image.

In the case of the recently activated terrestrial DMB (Digital Multimedia Broadcasting) video, one frame is composed of one slice while using H.264 compression. The slice unit compression method is designed to be robust to errors, but when an error occurs in one slice, decoding of the whole slice is impossible. This means that an error in a frame unit occurs.

An error generated in an image in which one frame is composed of one slice is treated as an error, and thus a more serious problem occurs in the image quality of the video compared to the existing pixel or block unit error. Therefore, there is a problem in that the previously developed block-based EC technique cannot be applied because the lost region is the entire frame.

For this reason, the conventional frame-based EC technique has generally used a frame freeze technique that outputs an image before loss in the form of a still image until a new instantaneous decoding refresh (IDR) screen is received.

2 is a diagram illustrating frames output after a frame error occurs when decoding a video using a general frame freeze technique. As shown in FIG. 2, since the frame freeze technique outputs a correctly transmitted frame before the frame is lost instead of the lost frame, the frame freeze technique may not accurately deliver image information to a user. If a large number of frames are lost and processed using the frame freeze technique, important information may be lost.

An object of the present invention is to provide an efficient frame loss concealment method and apparatus during video decoding to obtain high reconstruction efficiency.

According to one embodiment of the present invention for achieving this, the present invention provides a method for concealing frame loss during video decoding, comprising: checking a loss of an input frame during video decoding; In order to restore the frame in macroblock units, the macroblock at the same position as the macroblock to be restored and the neighboring macroblocks of the upper, lower, left, and right sides of the macroblock at the same position are used as the reference macroblock in the previously input frame. And setting a weighted average motion vector by examining each motion vector of the reference macroblocks and applying a preset weight to the checked motion vector, and using the weighted average motion vector. Restoring the macro block; Restoring any macroblock to generate a reconstruction frame that replaces the lost frame.

According to another aspect of the present invention, there is provided a frame loss concealment apparatus for decoding a video, comprising: a frame decoder which decodes a bit stream of an input encoded frame and checks whether a loss occurs for the decoded frame; A buffer unit which temporarily stores a previously decoded previous frame and a currently decoded current frame received from the frame decoder; A control unit which controls each component when performing the frame loss concealment function; A frame loss concealment unit generating a reconstructed frame similar to the original image by using a previously received frame when the input image frame is lost; An output unit which receives an input image frame without loss from the buffer unit and outputs the image frame as it is, or outputs a reconstructed frame generated by the frame loss concealment unit when a loss occurs in the input frame; The frame loss concealment unit includes a macro block at the same position as the macro block to be restored from a previously input frame to restore the input frame in units of macro blocks when a loss occurs in the input frame, and the macro at the same position. The upper, lower, left, and right neighboring macroblocks of the block are set as reference macroblocks to be used for reconstruction, and the weighted average motion vector is calculated by examining each motion vector of the reference macroblocks and applying a preset weight to the examined motion vectors. And restoring the lost macroblock using the weighted average motion vector, and restoring all the macroblocks of the lost frame to generate a reconstructed frame replacing the lost frame.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, specific details such as specific components are shown, which are provided to help a more general understanding of the present invention, and it is understood that these specific details may be changed or changed within the scope of the present invention. It is self-evident to those of ordinary knowledge in Esau. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

As a method of concealing the loss of a frame when decoding a video, the lost frame may be treated as the same frame as the previous frame, and the normal information received thereafter may be continuously decoded. Compared to the frame freeze method, this method can accurately convey the information about the image to the user, but many errors occur in the image frame that is output after the damaged frame, and the error is generated by Instantaneous Decoding Refresh (IDR). There is a problem that the screen continues to appear until the screen is received.

The frame loss concealment technique when decoding a video according to an embodiment of the present invention generates a new frame (hereinafter referred to as a reconstruction frame) that conceals a lost frame currently input using a previous frame normally received after the lost frame. This method minimizes the error propagated in the frame of the frame to enable normal decoding.

3 is a block diagram of a frame loss concealment apparatus when decoding a video according to an embodiment of the present invention. Referring to FIG. 3, a frame loss concealment apparatus according to an embodiment of the present invention includes a frame decoder 310 for decoding a bit stream of an input image frame and confirming whether or not a loss of the decoded image frame occurs. A buffer unit 330 for temporarily storing a previously decoded previous frame and a current decoded current frame received from the decoder, a controller 320 for controlling each component when performing a frame loss concealment function, and an input image frame The frame loss concealment unit 340 generates a reconstructed frame similar to the original image by using a frame normally received before the loss, and receives the input image frame without loss from the buffer unit 330 and outputs it as it is or without input. And an output unit 350 for outputting a reconstructed frame generated by the frame loss concealment unit 340 when a loss occurs in the lost frame.

Hereinafter, each configuration and operation of the frame loss concealment apparatus will be described in detail with reference to FIG. 3. An input bit stream for a video signal is input to the frame decoder 310. The frame decoder 310 decodes the data of the input bit stream according to an encoded format. This encoding format may be a general international standard such as H.263, H.264, MPEG-1, MPEG-2, MPEG-3, MPEG-4. The frame decoder 310 performs frame decoding according to this method, and outputs the decoded data to the buffer unit 330.

The buffer unit 330 stores a current frame storage unit 331 for storing an image frame input from the frame decoder 310 and data stored in the current frame storage unit 331 when a next frame is input. The previous frame storage unit 332 for receiving and storing the input. The frame stored in the previous frame storage unit 332 is stored for use as data for concealing a frame in which an error occurs.

In addition, the frame decoder 310 generates an error generation signal according to the decoding result of the frame and outputs the signal to the controller 320. The error generation signal according to the frame decoding result transmits information about the lost frame to the controller 320 when a frame is lost due to an error in a frame unit.

When the controller 320 receives an error generation signal indicating that an error has occurred in the frame received from the frame decoder 310, the controller 320 controls the buffer unit 330 and the frame loss concealment unit 340 to control the previous frame. Use macro blocks to create reconstructed frames to replace the failed frame. The controller 320 controls to replace the restored frame with a lost frame in which an error occurs, and outputs the restored frame from the output unit 350. However, when the frame is normally received, the controller 320 controls to directly output the received frame.

The previous frame storage unit 332 receives and stores the current image frame. If the frame input from the frame decoder 310 is a lost frame, the error output from the output unit 350 is concealed. In other words, a newly generated reconstructed frame may be stored to conceal a lost frame.

The frame loss concealment unit 340 generates a new reconstructed frame by referring to a previously input frame when a frame is lost due to an error in the input frame. Hereinafter, a method of generating a new reconstructed frame using a previously input frame will be described in detail.

4 is an exemplary view illustrating a method of referring to a macroblock of a previous frame of a frame loss concealment method when decoding a video according to an embodiment of the present invention. 4, the frame loss concealment method according to an embodiment of the present invention, in order to restore the macro block of the lost frame (Lost frame) according to the feature of the present invention of the same position of the previous frame (Previous frame) In addition to the macroblock, a method of restoring by referring to the motion vectors of the surrounding macroblocks of up, down, left, and right of the macroblock at the same position is also used. Here, the macro block at the same position as the lost macro block and the surrounding macro blocks on the upper, lower, left, and right sides are referred to as reference macro blocks.

Frame loss concealment method according to the present invention, according to a feature of the present invention, by applying a predetermined weight to the motion vector of the reference macroblocks of the previous frame to restore the lost macroblock, a motion vector (hereinafter, referred to as a weighted average) Motion block) and the weighted average motion vector is used to recover the lost macroblock. The weight is applied differently for each pixel of the lost macro block. In other words, it is applied differently according to the position in the block of the pixel to be reconstructed, so that the closer the macro block is to the pixel to be reconstructed, the higher the weight is. For example, suppose that you want to restore the leftmost pixel in a lost macro block, apply the weight of the motion vector of the closest left block among the reference macro blocks of the previous frame to be larger than the weight of the motion vector of the right block. Done. Hereinafter, the features of the present invention will be described in detail with reference to an embodiment of the present invention.

FIG. 5 is a diagram illustrating weights of motion vectors of respective macroblocks in a frame loss concealment method when decoding a video according to an embodiment of the present invention. 5 illustrates an embodiment of the present invention using a block of 4x4 pixels. Referring to FIG. 5, the center 4x4 rectangle represents a lost block, and the rectangles located on the top, bottom, left, and right sides represent the reference blocks of the previous frame. I and j represent the position of the pixel in the block, where i represents the horizontal axis and j represents the vertical axis, with the pixel at the top left as pixel (0,0) and the pixel at the bottom right as pixel (3,3 ).

In the frame loss concealment method according to an embodiment of the present invention, a weight of 15 is applied to a motion vector of a reference block at the same position of a previous frame using a block of 4x4 pixels, and (i + Apply a weight of 1), apply a weight of (j + 1) to the motion vector of the lower reference block, apply a weight of 5- (i + 1) to the motion vector of the left reference block, A weight of 5- (j + 1) is applied to the motion vector.

The equation representing the weight shows that the weight changes according to the position of the pixel having i and j values. Since the weight of the motion vector of the right reference block is i + 1, it can be seen that as the value of i increases, that is, as the position of the pixel approaches the right block, the weight increases. On the contrary, since the weight of the motion vector of the left reference block is 5- (i + 1), the weight is increased as the value of i becomes smaller, that is, the closer the position of the pixel is to the left block.

The weighted average motion vector calculated for each pixel is expressed as follows.

In the above equation, mv _C , mv _T , mv _B , mv _L and mv _R denote motion vectors of reference blocks located at the center, top, bottom, left and right, respectively.

The weight is chosen as the most efficient value through the experiment, applying 60% weight to the motion vector of the centered reference block, and depending on the position of the pixel to restore the remaining 40% weight, the up, down, left, and right reference blocks Applies to The weights applied to the up, down, left, and right reference blocks apply more weights to the closer blocks according to the proximity of the pixels to the blocks.

The loss concealment method of the macroblock used in the present invention can be applied to a block of 4x4 pixels as in one embodiment of the present invention, but using the same principle as applied to a block of 4x4 pixels, The same applies to macro blocks.

The frame loss concealment unit 340 applies the above-described macroblock restoration scheme to the lost frames as a whole to restore all macroblocks, thereby generating a restoration frame that replaces the lost frames.

6 is a flowchart illustrating a frame loss concealment operation when decoding a video according to an embodiment of the present invention. Referring to FIG. 6, when an image frame is input in step 602, the process proceeds to step 604 to decode the image frame in the frame decoder 310. If the frame is lost due to an error in the frame after determining whether an error frame has occurred in step 606, the process proceeds to step 608 in which the lost frame in which the error occurred using the previous frame is framed in units of macroblocks. Reconstruct to create reconstructed frames to replace lost frames. In operation 610, the restored frame is output. If no error occurs in the frame in step 606, the process proceeds directly to step 610 and outputs the corresponding frame. If it is determined in step 612 that the operation is to be terminated, the operation is terminated. If not, the operation returns to step 602 to repeat the operation.

7 is an exemplary diagram of frames output after a frame error occurs when decoding a video using the frame loss concealment method according to an embodiment of the present invention. FIG. 7 is an image of error recovery after frame loss using the method proposed by the present invention, which shows high recovery efficiency compared to FIG. 1 and shows excellent performance compared to the frame freeze method of FIG. have.

As described above, an operation and a configuration of a method and apparatus for concealing a frame loss when decoding a video according to an embodiment of the present invention can be made. Meanwhile, in the above description of the present invention, a specific embodiment has been described. It can be carried out without departing from the scope of. Therefore, the scope of the present invention should not be defined by the described embodiments, but by the claims and equivalents of the claims.

According to the present invention, when a video error occurs and loss occurs on a frame-by-frame basis, the video loss can be restored more efficiently, and the error can be prevented from propagating to a frame transmitted later.

Claims (8)

In the method of concealing frame loss when decoding a video, Checking the loss of the input frame during video decoding; If the input frame is a lost frame, the macroblock at the same position as the macroblock to be restored from a frame previously input for each macroblock to restore the lost frame in units of macroblocks; Setting neighboring macroblocks on the top, bottom, left, and right sides of the same macro block as the reference macro block to be used for restoration; Checking each motion vector of the reference macroblocks and calculating a weighted average motion vector by applying a preset weight to the checked motion vector; Restoring the macroblock to be restored using the weighted average motion vector; Restoring all macroblocks of the lost frame to generate a restored frame to replace the lost frame. The method of claim 1, wherein when applying weights to the motion vectors, Apply a predetermined ratio weight to a motion vector of a reference macro block at the same position as the macro block to be restored among the reference macro blocks, The remaining weights are applied to the motion vectors of the up, down, left, and right reference macro blocks in proportion to the proximity of the up, down, left, and right reference macro blocks to positions of pixels to be reconstructed. The method of claim 2, wherein a weight of 60% is applied to a motion vector of a reference macro block at the same position as the macro block to be reconstructed. The method of claim 1, further comprising outputting the input frame as it is when the input frame is a lossless frame. The method of claim 1, wherein the weighted average motion vector is calculated using the following Equation if the macro block is a block composed of 4x4 pixels. Equation

In the above equations, i and j denote positions of pixels in a block, and mv (i, j) denotes a weighted average motion vector of pixel (i, j), and mv _C , mv _T , mv _B , mv _L and mv _R are the motion vectors of the reference macro block in the middle, top, bottom, left and right, respectively. In the frame loss concealment apparatus when decoding a video, A frame decoder which decodes a bit stream of an input encoded frame and checks whether a loss of the decoded frame occurs; A buffer unit for temporarily storing a previously decoded previous frame and a currently decoded current frame received from the frame decoder; A control unit for controlling each component when performing the frame loss concealment function; A frame loss concealment unit for generating a reconstructed frame similar to the original image by using a previously received frame when the input image frame is lost; And an output unit for receiving an input image frame without loss from the buffer unit and outputting the image frame as it is or outputting a reconstructed frame generated by the frame loss concealment unit when a loss occurs in the input frame. The frame loss concealment unit may include a macro block at the same position as a macro block to be restored from a previously input frame for each macro block to restore the lost frame in units of macro blocks when the input frame is a lost frame; The upper, lower, left, and right peripheral macroblocks of the same macroblock are set as reference macroblocks to be used for reconstruction, and each motion vector of the reference macroblocks is examined and weighted by applying a preset weight to the examined motion vectors. Calculating an average motion vector, restoring the macroblock to be restored using the weighted average motion vector, and restoring all the macroblocks of the lost frame to generate a recovery frame replacing the lost frame; Frame loss concealment device. 7. The apparatus of claim 6, wherein the buffer unit comprises a current frame storage unit for storing the current frame received from the frame decoder, and a previous frame storage unit for storing the previous frames. The apparatus of claim 6, wherein the controller controls to output the received frame immediately when the frame is normally received.

Images