KR100228684B1 - A temporal predictive error concealment method and apparatus based on the motion estimation - Google Patents

Abstract

The present invention relates to a method and apparatus for concealing time prediction error based on motion estimation to estimate a motion vector for a peripheral partition except the damaged partition and to reduce the discontinuity between the damaged partition and the surrounding partition restored by the vector. The present invention includes a motion estimating unit that sets a region consisting of a damaged section and adjacent neighboring sections, and calculates a difference between a target portion in a search region in a reference frame and a remaining region except the damaged section in the set region. . The error concealment unit reads the image corresponding to the damaged partition position designated by the motion vector from the image in the reference frame, replaces it with the image value of the damaged partition, and conceals the error by reducing the discontinuity between the recovered damaged partition and the surrounding partitions. Therefore, there is an effect that can improve the subjective / objective image quality of the transmitted image.

Description

Method and apparatus for concealing time prediction error based on motion estimation

The present invention relates to a method for concealing an error generated during video transmission. In particular, the present invention relates to a method of concealing an error vector in a neighboring section except for a damaged section, and to reduce the discontinuity between the damaged section and the neighboring section recovered by the vector, thereby reducing the error at the receiving end. A method and apparatus for concealing time prediction error based on motion estimation that is not detected.

In recent years, research and development for digital video / audio services have been actively conducted worldwide. In particular, efforts have been made not only to transmit information in one direction such as conventional television but also to provide various services in two directions through communication channels. For example, high-definition television (HD-TV) with large screen and stereoscopic sound can make you feel the impression in a movie theater, and you can order and enjoy the TV program, music, and games you want (on-demand video / audio / game). ). You can talk to people who are far away through video calls or video conferencing, make bank payments at home (home banking), and purchase or treat while looking at product information or patient status on a computer screen (home shopping, remote medical care). ). In addition, commerce documents and video information between companies are electronicized (e-data exchange, electronic transactions, databases), and school lectures are held at home (remote lectures), and work from home is possible.

At this time, the information has various forms such as text, figure, voice, sound, still image, video, and general data, and is mainly stored or transmitted in the form of multimedia combined mainly with image and sound. If they are processed by a simple modulation technique, the amount of data becomes excessive, which causes a lot of difficulties in storage or transmission. In particular, the use of a limited communication channel requires a high compression rate, that is, a technique for effectively compressing data in order to maximize data transmission efficiency. However, if an error occurs in the information with high compression rate, important information may be lost or wrong information may be transmitted, which may have a fatal effect on the receiving system. For example, information may be distorted or inter-symbol interference may occur due to non-ideal characteristics of the transmission channel, that is, noise, nonlinear filtering, multipath, and Doppler shift.

In order to compensate for such an error, an error correcting code has been added, and a trellis coded modulation technique has been introduced. In recent years, it has been extended to the case of using a noisy transmission path such as a wireless channel in the city, and therefore, it is important to prepare a countermeasure in case of an error. A conventional MPEG (Moving Picture Experts Group) decoding apparatus decodes a bit stream transmitted by compression encoding and then restores original data using the image data and a motion vector transmitted from the encoding apparatus. At this time, if an error occurs while passing through the transmission channel, the image of the damaged section is recovered so that the receiver does not feel that an error has occurred.

To do this, first, the image of one frame is subdivided into several partition units, and the damaged sections are detected from these frames to obtain a motion vector for the partition. The vector is used to find the partition that most closely resembles the damaged partition in the reference frame, and then replace the damaged partition with the image. As such, the motion vector used to recover the image of the section in which the error occurred is obtained using the section adjacent to the damaged section. That is, the motion vector of the damaged partition is obtained by averaging the motion vectors of the surrounding partitions (for example, all or some of the partitions located in the up / down / left / right / diagonal directions) surrounding the damaged partition. Also, a median filter is used to take a median of the motion vectors of the neighboring sections and then use the median filter.

However, this conventional method has a problem in that it cannot be utilized when the motion vector does not exist in the peripheral partitions surrounding the damaged partition. For example, since the motion vector is information representing a portion most similar to the current frame in the reference frame, the motion vector cannot be utilized when the motion information is not transmitted because it is encoded as Intra.

The present invention is to solve the above-mentioned conventional problems, an object of the present invention is to replace the image damaged by the error occurred when compressing and transmitting the image to the adjacent image on the time axis to improve the quality of the transmitted image The present invention provides a method and apparatus for concealing time prediction error based on a motion estimation.

Another object of the present invention is to estimate a motion vector for a neighboring partition except for a damaged partition, and to reduce the discontinuity between the damaged partition and the neighboring partition recovered by the vector, thereby predicting a motion based on the motion estimation based on the motion estimation to prevent an error from being detected at the receiving end. To provide a method and apparatus for concealment.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram showing a schematic configuration of a video data decoding apparatus employing a time prediction error concealment method based on motion estimation of the present invention.

2 is a screen state diagram showing an example of a method for estimating a motion vector for a damaged section in the motion estimation unit of the present invention;

3 is a screen state diagram showing a region consisting of damage sections and peripheral sections having an arbitrary shape;

※ Explanation of code for main part of drawing

5: frame memory 6: motion compensator

7: motion estimation unit 8: error concealment unit

In order to achieve the above object, a time prediction error concealment method based on the motion estimation of the present invention sets a region consisting of a damaged partition and adjacent neighboring partitions, and includes a target portion and a set region in a search region in a reference frame. The first step of estimating the motion vector by calculating the difference between the remaining regions except the damaged section, and reading the image corresponding to the damaged partition position specified by the motion vector from the reference frame image, replacing the image with the damaged section The second step is to conceal the error by reducing the discontinuity between the damaged and surrounding compartments.

In addition, the time prediction error concealment apparatus based on the motion estimation of the present invention for achieving the above object includes a motion estimation unit for estimating a motion vector by reading the target portion on the search region of the reference frame and the set region from the frame memory. The motion estimating unit sets a region consisting of a damaged section and adjacent neighboring sections, and calculates a difference between the target portion in the search region in the reference frame and the remaining region except the damaged section in the set region. The motion vector of the motion estimation unit is input to the error concealment unit. The error concealment unit conceals the error by reading the image corresponding to the damaged partition location specified by the motion vector from the reference frame image and replacing it with the image value of the damaged partition, and reducing the discontinuity between the recovered damaged partition and the surrounding partitions.

Hereinafter, an embodiment of the present invention will be described in detail with reference to FIGS. 1 and 2 as follows.

1 is a block diagram showing a schematic configuration of a video data decoding apparatus employing a time prediction error concealment method based on motion estimation according to the present invention. Reference numeral 10 is a decoding apparatus which is commonly used, and the present invention is a form in which a time prediction error concealment function block is added to an output terminal of the decoding apparatus 10. Sign 1 is a variable-length decoder that receives image data made of one sign by tying a run (number of consecutive zeros) and a level (any non-zero value immediately following the consecutive zeros). The variable length decoder 1 decodes the data input in the encoded state through an inverse process of encoding to extend the image data. An inverse quantizer 2 is connected to an output terminal of the variable length decoder 1 to inversely quantize image data into a transform coefficient of a frequency domain according to a quantization level signal transmitted from an encoding apparatus. The transform coefficients output from the inverse quantization unit 2 are input to an inverse DCT (Discrete Cosine Transform) unit 3, where inverse discrete cosine transforms into image data of a spatial domain.

At this time, the video data inversely transformed by the inverse DCT unit 3 is reproduction error data corresponding to the error data calculated by the encoding apparatus, and this data is input to the adder 4. Meanwhile, the decoding apparatus of the present invention includes a frame memory 5 which stores image data fed back in the decoded state on a frame basis. The motion compensator 6 and the motion estimator 7 are respectively connected to the output end of the frame memory 5, and at this time, the motion vector transmitted from the motion estimator of the encoder is input to the motion compensator 6. The motion compensator 6 reads block data having a predetermined size corresponding to the motion vector transmitted from the frame data stored in the frame memory 5 and supplies it to the adder 4. The adder 4 adds the image data output from the inverse DCT unit 3 and the block data output from the motion compensator 6 to output the reconstructed image data to the error concealment unit 8. At this time, the used reconstructed image data (output data of the adder 4) to be used as a reference frame is stored in the frame memory 5 for future motion compensation.

The features of the invention are the motion estimator 7 and the error concealment 8. The motion estimating unit 7 sets a region consisting of a damaged section and neighboring sections adjacent thereto, and by means of an error concealment section 8 and a target portion in the search region in the reference frame read from the frame memory 5. The difference between the set areas inputted is calculated. When estimating the motion vector as described above, the difference between the regions generated by the damaged partition or the neighboring partition that caused the error is excluded from the calculation. The motion estimation unit 7 sequentially inputs the position of the target part in the search area into the frame memory 5 to receive the corresponding part data, and the error concealment unit 8 receives area data in which the difference with the set area is minimized. Will output

The error concealment unit 8 replaces the image corresponding to the damaged partition position designated by the estimated motion vector among the images in the input reference frame with the image value of the damaged partition, and reduces the discontinuity between the recovered damaged partition and the surrounding partitions. Cover up. Typically, image data (output data of the adder 4) reconstructed for use as a reference frame is stored in the frame memory 5. However, as another embodiment of the present invention, as indicated by the dotted line in FIG. 1, the output data of the error concealment section 8, which has been completed until the error concealment process, is used as a reference frame, and is configured to be stored in the frame memory 5 can do. In such a method, there is a problem that the reference frame is different between the encoding apparatus on the transmission side and the decoding apparatus on the receiving side. However, such a configuration may be useful when there are particularly many errors in the transmission data.

In the image data decoding apparatus operating as described above, a time prediction error concealment method based on the motion estimation performed by the motion estimation unit 7 and the error concealment unit 8 will be described.

Temporal prediction error concealment is a technique to fill in the damaged part by taking the data at the proper position in the reference frame. To this end, a damaged portion must first be detected. Although not illustrated in the decoding apparatus of FIG. 1, an error detection circuit receives an image code string transmitted from an encoding apparatus and detects a damaged portion. The error concealment section 8 finds and replaces in the reference frame the section most similar to the original information of this section for the detected damaged section. In this case, the original information can be reproduced almost completely with respect to the lossless part of the screen, especially the background part.

However, for the moving part, the most similar part of the current damaged part of the reference frame data should be taken. To this end, a motion vector representing a portion of the reference frame most similar to the corresponding portion of the current frame is required. However, if a special transmission structure for transmitting a motion vector is not used, the motion vector is also lost when an error occurs. Therefore, the decoder must estimate the motion vector of the damaged partition again.

2 is a screen state diagram illustrating an example of a method of estimating a motion vector for a damaged section in the motion estimation unit 7 of the present invention. In FIG. 2, the damaged section (a) is shown as a rectangle for convenience of description, but may generally have any shape. First, the area b containing the peripheral compartments adjacent to the damaged compartment a is set. Next, the set area b, that is, the N partitions surrounding the damaged partition a, are grouped into one area to estimate the motion with respect to this area b. At this time, the portion of the damaged compartment (a) or the neighboring compartments, which is an error compartment, is excluded from the calculation. Normally N is assumed to be 8, i.e., eight peripheral compartments each located in the up, down, left, right and diagonal directions of the damaged compartment (a), but are not necessarily limited to eight. In addition, the damaged compartment (a) or the peripheral compartment (b) does not necessarily have to be rectangular, but may have any shape as shown in FIG. 3. In this case, the arbitrary form has a form of one object which is a coding unit.

In order to estimate the motion vector of the region b set as described above, a difference, that is, a correlation between the target portion in the search region in the reference frame and the given region b, is calculated. Here, the "difference" is a unit of measurement for indexing the similarity between the two regions, and is calculated using a calculation formula such as "sum sum squared difference" or "sum sum absolute difference". Thus, when calculating the difference between the given areas, except for the difference between the areas caused by the damaged partition (a) in the given area (b), the remaining values are calculated. The vector can be estimated.

The reference frame used in the time prediction error concealment apparatus of the present invention is an image that is later or later in time than the currently decoded image, and is typically data that has not been subjected to time prediction error concealment or other post-processing after decoding. In this case, the reference frame is the same as that used on the transmitting side. However, as another embodiment of the present invention, data passed through time prediction error concealment after image decoding or other post-processing for image quality improvement may be used as a reference frame. In this case, the reference frame stored in the receiving side may be different from that of the transmitting side, but there is an advantage in that it is possible to avoid the case where the error of the reference frame itself is poor, especially when there are many errors or poor image quality.

Since the image information of neighboring partitions including damaged partitions in a normal image is not significantly changed compared to the reference frame, even if the motion estimation is performed using only the information of the neighboring partitions, a motion vector almost similar to the original motion vector can be calculated. When estimating the motion vector as shown in FIG. 2, the reference frame may be temporally ahead (T1> T2) or behind (T2> T1) of the current video frame, and must be available before using the current frame. There is no limit to that.

The error concealment 8 of FIG. 1 uses the motion vector obtained in this way to recover the damaged compartment. That is, the image corresponding to the damaged partition position designated by the estimated motion vector in the reference frame image is read and replaced with the image value of the damaged partition. In this case, there may be a discontinuity between the repaired damaged compartment and the surrounding compartments. First, the original image value (denoted as A), which is normally restored by the decoding apparatus of FIG. 1, is obtained for the region b adjacent to the damaged damage section. Then, an image value (denoted as B) corresponding to the region (b) adjacent to the damaged partition position designated by the estimated motion vector in the reference frame image is obtained. Discontinuity can be reduced as much as possible by substituting the image value of the area (b) with the value obtained by processing the two image values obtained (marked C).

There are several ways to process two image values, the simplest being the weighted sum of A and B.

C = weight × A + (1-weight) × B

Here, the weight may vary depending on the position of the pixel. For example, the weight is reduced from the damaged section (a) to the edge of the adjacent section (b) to reduce the discontinuity. It is also possible to change the shape of the surrounding compartments included in the calculation. In other words, the calculation amount may be reduced by performing the calculation only on the pixels of the adjacent part instead of all the eight surrounding areas.

As described above, the present invention replaces and repairs a damaged image by an error occurring when compressing and transmitting a video to an adjacent image on a time axis, thereby eliminating adverse effects on a receiving system and significantly improving the subjective and objective image quality of the transmitted image. have. In addition, the motion vector is estimated for the neighboring partitions except the damaged one, and the discontinuity between the damaged and neighboring partitions restored by this vector is reduced so that the receiver cannot detect errors. It can be used widely.

Claims (14)

A method for detecting a damaged section while decoding a transmitted video bit stream, (1) estimating a motion vector by setting a region consisting of a damaged section and adjacent neighboring sections and calculating a difference between a target portion in the search region in the reference frame and the remaining region except the damaged section in the set region; ; And (2) reading the image corresponding to the damaged partition position designated by the motion vector from the reference frame image, replacing the image with the damaged partition image value, and concealing errors by reducing the discontinuity between the repaired damaged partition and the surrounding partitions. A method for concealing time prediction errors based on motion estimation. The method of claim 1, wherein the neighboring partitions are all eight partitions or a partial partition of eight partitions located in the up, down, left, right, and diagonal directions of the damaged partition. . The method of claim 1, wherein the difference between the target portion in the search region in the reference frame and the remaining region except the damage section in the set region is calculated by using a calculation expression indicating “sum of squared errors” or “sum of absolute errors”. A time prediction error concealment method based on motion estimation, characterized by the above-mentioned. 2. The method of claim 1, wherein the reference frame is an image that is earlier or later in time than the current image frame among images stored in units of frames. 5. The method of claim 4, wherein the reference frame is image data after decoding a transmitted image bit stream and before performing post-processing for error concealment or image quality improvement. 5. The method of claim 4, wherein the reference frame is image data after decoding the transmitted image bit stream and performing post-processing for error concealment or image quality improvement. The method of claim 1, wherein the original image value restored by the decoding apparatus and the image value corresponding to the set region designated by the motion vector in the reference frame image are read and replaced with the image value of the set region. A method for concealing time prediction errors based on motion estimation, characterized by performing a process of reducing discontinuity between the repaired damaged compartment and surrounding compartments. 8. The method of claim 7, wherein the weighted sum value C is obtained based on the following equation when the two image values are processed. C = weight × A + (1-weight) × B Here, A is the original image value reconstructed by the decoding apparatus for the set region, B is an image value corresponding to the set region among the images in the reference frame designated by the motion vector, and weight is a weight. 9. The method of claim 8, wherein the weight is varied according to the position of the pixel. 2. The method of claim 1, wherein the partition is of any shape. 11. The method of claim 10, wherein the partition is a rectangle of a predetermined size. In the apparatus for decoding video data, A motion estimating unit configured to set a region consisting of a damaged section and adjacent neighboring sections, and read a reference frame from a frame memory to calculate a difference between a target portion in the search region and the remaining region except the damaged section in the set region; And An error concealment unit which reads an image corresponding to a damaged partition position designated by the estimated motion vector from a reference frame image and replaces it with an image value of the damaged partition and conceals errors by reducing discontinuities between the repaired damaged partition and neighboring sections. 2. A time prediction error concealment apparatus based on motion estimation, characterized in that. The time prediction based on the motion estimation according to claim 12, wherein the frame memory stores an image for use as a reference frame among image data after decoding and before post-processing for error concealment or image quality improvement. Error concealment device. The time prediction based on motion estimation according to claim 12, wherein the frame memory stores an image for use as a reference frame among image data after the error concealment processing or the post-processing for image quality improvement. Error concealment device.

Images