KR100801155B1 - A spatial error concealment method with low complexity in h. 264/avc video coding - Google Patents

Abstract

A spatial error concealment method having low complexity in H.264 is provided to restore a lost macro block quickly only by using a less operation amount, thereby decoding superior image quality in real time even when the macro block within an intra frame is lost. A spatial error concealment method having low complexity in H.264 comprises the following steps of: predicting a direction of an edge of a lost macro block by selecting a prior prediction mode among prediction modes of neighboring blocks located around the lost macro block; dividing an area within the lost macro block into an edge area where there is an edge and a flat area where there is no edge by using the selected prior prediction mode; restoring the edge area through an edge based directional interpolation technique, and restoring the flat area through an interpolation technique using a weighted average to restore the lost macro block.

Description

A spatial error concealment method with low complexity in H.264 / AVC video coding

1 is a diagram showing four prediction modes of intra16x16.

2 is a diagram illustrating nine prediction modes of intra 4x4.

3 is a block diagram of a procedure of a spatial error concealment method according to an embodiment of the present invention.

4 is an understanding diagram showing four pixel values perpendicular to the prediction direction in each intra mode used for calculating an edge size value according to an embodiment of the present invention.

FIG. 5 is an exemplary diagram for selecting an edge region and a flat region when prediction mode 4 is selected as the dominant prediction mode and B ₁ , B ₅ , and B ₁₀ have prediction mode 4.

6 is a diagram showing an interpolation method in an edge direction for restoring pixel values in an edge region.

FIG. 7 is a comparison table of calculation amounts for one lost macroblock when no neighboring block having prediction mode 2 exists and the edge region occupies 50% of the total.

8 is a comparison table of the objective image quality of the method in the reference software, the Alkachouh method and the error concealment method according to the present invention at different bit rates.

FIG. 9 shows the first frame of the Foreman sequence when the value of the quantization coefficient is 20 (a) 20% macroblock loss rate (PSNR = 11.26dB) (c) The image reconstructed using the method of the reference software ( PSNR = 29.82dB) (d) Image reconstructed using Alkachouh's method (PSNR = 28.27dB) (e) A diagram showing an image reconstructed using the error concealment method according to the present invention (PSNR = 34.85dB).

The present invention relates to a spatial error concealment method having a low complexity in H.264, in which the prediction mode information of the intra block is significantly related to the edge direction in the block, and thus the lost macroblock in the intra frame The estimated macro direction of the lost macroblock is predicted using the prediction mode information of a block located near the eigen, and the lost macroblock is separated into an edge region and a flat region, and then different interpolation techniques are used. A spatial error concealment method having a low complexity in H.264 which can restore a block to low complexity.

The H264 / ACC standard was established in collaboration with ITU-T's Video Coding Experts Group (VCEG) and ISO / IEC's Moving Picture Expert Group (MPEG). The H.264 / AVC video coding scheme adds new coding tools that were not found in the previous video coding standards (MPEG2, MPEG4 Part 2, H.263). In comparison, it has a higher compression efficiency.

Newly added coding tools are as follows.

First, H.264 performs integer conversion of 4 × 4 units, and there is no mismatch between the encoder and the decoder. This is compared with several existing video coding standards that had a problem of mismatch between encoder and decoder by using discrete cosine transform of 8 × 8 units. In addition, when a macroblock is selected as an intra 16 × 16 mode, DC values of a 4 × 4 transform block, which is a transform performing unit, perform a Hadamard transform once again, and through this hierarchical transform, High compression efficiency. In addition, the quantization parameter in H.264 is used as one index indicating quantization data and has a value of 52 steps from 0 to 51.

Second, in H.264, two entropy coding methods are used. In other words, the entropy coding technique converts most of the syntax except the transform coefficients into one codeword table according to the characteristics of the data, and then encodes it into EGC (Exponential Golomb Code). There is a method of encoding by CAVLC (Context Adaptive Variable Length Coding) and a method of adaptively selecting estimated symbol probability distribution models by adding context modeling to an existing arithmetic coding scheme.

Third, block-based motion estimation, transformation, and quantization techniques bring many advantages to the encoder design, but involve blocking artifacts. H.264 eliminates blocking by using loop filters in the encoding and decoding process to efficiently improve such blocking.

Fourth, the H.264 coding scheme includes intra prediction codes and processes in the spatial domain. The prediction coding method in space is performed with 16 × 16 size and 4 × 4 size for the luminance component, and has a prediction mode in consideration of four and nine directions in space, and the color difference component has a luminance of 16 × 16 size. It has four prediction modes as component. Intra prediction of 16 × 16 size and 4 × 4 size in the spatial domain with respect to the luminance component has high compression efficiency in the detailed region as well as the flat region of the image.

Fifth, in H.264, previously reconstructed frames are used as reference pictures. The use of such a multi-reference image has a higher compression efficiency than the case of using only a single reference image, and is more prominent in the case of encoding an area that is covered by a repetitive image or another object.

Sixth, in H.264, the size of the block is varied and the motion is estimated to be 16 × 16, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, and 4 × 4. This contrasts with H.263 and MPEG-4 technology, which previously compensated for motion only in 16 × 16 and 8 × 8 sizes. Motion estimation of block sizes of various sizes, especially 8 × 4, 4 × 8, and 4 × 4 blocks, which are less than 8 × 8 sub macroblocks, can achieve high compression efficiency even in more detailed image areas. To be.

In addition, coding tools such as adaptive frame / field coding and network abstraction layer (NAL) have been added. These newly added coding tools enable the H.264 coding standard to have higher compression efficiency than previous video coding standards. In the case of encoding the main profile using five reference frames and a 32-pixel motion estimation region, the H.264 standard is known to reduce the bit rate by about 48% to 78% compared to MPEG-2. Thomas Wiegand et al., "Rate-Constrained Coder Control and Comparison of Video Coding Standard."

Among these, various block-size motion estimation techniques require more computation compared to existing coding standards, but are a coding tool that significantly affects coding efficiency by estimating motion in more granular units.

Another feature of estimating motion using these various block sizes is that it generates more motion vectors than existing video standards. That is, while the existing video standard has only one motion vector for one macroblock having a size of 16 × 16, H.264 can have a maximum of 16 motion vectors for one macroblock of size 16 × 16. . This indicates that the amount of motion vector information of neighboring macroblocks that can be used in error concealment in H.264 is larger than that of the existing coding standard.

In addition, by using several reference images, each macroblock includes information of a reference frame together with a motion vector. Accordingly, one macroblock in H.264 includes up to 16 different motion vectors, up to 4 reference frame information, and macroblock mode information defining a unit of motion estimation.

Since MPEG-2 was enacted, much research has been done on visual error concealment. Keiu proposed a technique for concealing errors by copying a reference block using an estimated motion vector by estimating the motion of a lost macroblock [pp. Of "Cell-loss concealment techniques for layered video codecs in an ATM network". 666-677]. However, when the estimation of the motion vector is not accurate, there is a drawback that the difference between the neighboring blocks is large.

In a similar way, Suh and Kwon, pp. 750-753 and mean values of surrounding macroblock motion vectors, respectively, are described in pp. 203 of “Error Concealment Techniques for H.264 Video Transmission”. 276-279], a technique for concealing errors is proposed. It must be assumed that the missing macroblock and neighboring macroblocks have a high correlation between motion vectors.

Jung has proposed a modified BMA method that improves the BMA (Boundary Matching Algorithm) method for estimating motion vectors using neighboring boundary pixels of damaged macroblocks. See "Robust error concealment algorithm using iterative weighted boundary matching criterion." 384-387]. However, this has the disadvantage of having a high complexity.

The various forward error concealment schemes above are suitable for previous coding standards for performing motion estimation in a 16 × 16 size. Zheng concealed the error by reconstructing the motion vector pixel-by-pixel using more motion vector information in H.264, which estimates motion with various block sizes, but this also does not consider multi-reference frame and macroblock mode information. [A motion vector recovery algorithm for digital video using lagrange interpolation ”pp. 383-389.

The occurrence of an error in an image during transmission of image data through a network that may include an error causes a significant deterioration in image quality. In order to maintain a constant image quality of a compressed image in a transmission environment in which such an error exists, an error control technique such as an error tolerance technique and an error correction technique is required. In particular, the error concealment technique that can be implemented independently at the receiving end is an important technique for obtaining high quality images.

The error concealment technique can be largely divided into a spatial error concealment technique and a temporal error concealment technique. A spatial error concealment technique is a technique for recovering an error of an image by using correlation between pixel values in a screen. It is a technique of restoring an error of an image by using a correlation between a previously received screen and a current screen.

Spatial Error Concealment In particular, as an existing method for error concealment in the intra frame of video, Wang has proposed an optimization algorithm that estimates the optimal DCT coefficient using smoothness constraint between adjacent pixels. Et al. Proposed a block reconstruction algorithm using an interpolation filter and fuzzylogic reasoning in the spatial domain.

A block restoration algorithm based on POCS (projection onto convex sets) was proposed by Sun and Kwok, and an interpolation technique based on fast DCT was proposed by Alkachouh and Bellanger. Recently, a restoration technique called RIBMAP (recovery of image blocks using the method of alternating projection) has been introduced by Park et al.

However, the above-mentioned spatial error concealment techniques are applicable to both still images and moving images, which do not take into account the encoding characteristics of the object to be applied.

The present invention was devised to solve the above problems of the prior art, and the prediction mode information of an intra block is significantly related to the edge direction in the block. Predict the edge direction of the lost macroblock using the prediction mode information of the block located at, and separate the lost macroblock into the edge region and the flat region, and then use different interpolation techniques. It is an object of the present invention to provide a spatial error concealment method having a low complexity in H.264, which can be restored with a small amount of computation and a low complexity.

In order to achieve the above technical problem, the constituent means of the spatial error concealment method having a low complexity in the present invention H. 264 is a dominant prediction mode among prediction modes of neighboring blocks located near the lost macroblock. Estimating the direction of the edge of the lost macroblock by classifying and classifying the region within the lost macroblock into the edge region where the edge exists and the planar region where the edge does not exist using the selected predominant prediction mode. And restoring the lost macroblock by restoring the edge region by an edge-based directional interpolation technique and restoring the flat region by an interpolation technique using a weighted average.

In addition, the edge direction of the lost macroblock is predicted by using the prediction mode for the 16 4x4 blocks located in the top, bottom, left and right of the lost macroblock and the size of the edge in each block.

In addition, the size of the edge of each 4x4 block selects four pixel values in which each block is perpendicular to the prediction direction of the prediction mode used in the encoding process, and the maximum value among the four selected pixel values. And is determined by the difference between the minimum value.

Further, the dominant prediction mode is determined as a prediction mode in which the sum of edge sizes in each prediction mode is the maximum among the prediction modes in the 16 4x4 blocks, and the prediction direction of the dominance prediction mode is the lost macroblock. It is characterized by determining in the direction of the inner edge.

The edge region may select a block having the selected predominant prediction mode among neighboring blocks located around the lost macroblock, and the selected block is an area present in the prediction direction of the selected predominant prediction mode. It is characterized by determining. In this case, the edge region preferably further includes a margin of a predetermined region.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the spatial error concealment method having a low complexity in the H.264 of the present invention consisting of the above configuration means.

The present invention uses the prediction mode information contained in the H.264 bit string to recover the lost macroblocks in the intra frame. The H.264 coding standard has three intra modes to encode intra frames: intra16x16, intra8x8 and intra4x4, each of which has a different number of prediction modes.

1 and 2 show a prediction mode of intra16x16 and intra4x4 modes, and intra8x8 has the same prediction mode as intra 4x4.

The present invention is largely composed of two processes as shown in FIG. The first process is pre-processing, which is to estimate the edge direction in the macroblock by selecting the dominant prediction mode (DPM) among the prediction modes around the lost macroblock. It is composed of the process of classifying the area in the macroblock into the edge area where the edge exists and the planar area where the edge does not exist using the selected predominant prediction mode.

The second process is to recover the lost macroblock using the actual interpolation technique. The edge and planar regions of the lost macroblock are restored using different interpolation techniques.

In the pre-processing, the direction of the edge in the macroblock is estimated using the prediction mode for 16 4x4 blocks located on the top, bottom, left and right of the lost macroblock. In this case, if one or more of the neighboring macroblocks are encoded as Intra16x16 or Intra8x8, it is considered that the same intra4x4 prediction mode is repeatedly displayed.

For example, if one macroblock is encoded in the intra16x16 mode and prediction mode 1 is used, it is assumed that all 16 4x4 blocks existing in the macroblock have prediction mode 1. This is because the prediction directions of the prediction modes 0 to 3 in intra16x16 have the same prediction direction as those of the prediction modes 0 to 3 in intra4x4.

Meanwhile, in order to estimate the edge direction in the lost macroblock, the edge size as well as the prediction mode must be considered. The size of the edges in the 16 neighboring 4x4 blocks is determined by selecting four pixel values perpendicular to the prediction direction of the prediction mode used in the encoding process, and selecting the maximum and minimum values among the four selected pixel values. Obtained by the difference, the attached Figure 4 and the following equation (1) represents this. As shown in Equation (1) below, in the prediction mode 2, it is determined that there is no edge, and the size of the edge is zero.

식 (1)

Formula (1)

When the edge sizes of 16 4x4 blocks located around the lost macroblocks are obtained, the prediction mode in which the sum of the edge sizes in each prediction mode is the largest among the prediction modes in these blocks is selected as the dominant prediction mode. The prediction direction of the dominant prediction mode is selected as the direction of the edge in the lost macroblock.

Therefore, the present invention may consider the same edge direction as the prediction direction of the prediction modes except the prediction mode 2, and the edge direction of the lost macroblock is 0 ° when each prediction mode is selected as the dominant prediction mode (prediction mode 1). , 22.5 ° (prediction mode 8), 45 ° (prediction mode 3), 67.5 ° (prediction mode 7), 90 ° (prediction mode 0), 112.5 ° (prediction mode 5), 135 ° (prediction mode 4), 157.5 ° (prediction mode 6).

The second step of the present invention is a process of dividing a region in a lost macroblock into an edge region and a flat region. An edge region refers to a region where edge components exist in a macroblock, and a flat region refers to a region where no edge exists.

The edge region selects a block having a prediction mode selected as the dominant prediction mode among the 16 4x4 blocks located around the lost macroblock, and selects the region in the prediction direction of the dominant prediction mode from the selected block. At this time, the margin area is widened to increase the width of the edge area because the edge of the natural image varies in value over several pixels unlike the ideal edge.

An area not selected as an edge area after the edge area is selected is considered to be a flat area. 5 shows an example of edge region selection when prediction mode 4 is selected as the dominant prediction mode. As shown in FIG. 5, if the blocks B ₁ , B ₅ , and B _{10 have} the same prediction mode 4 as the dominant prediction mode, the prediction direction of the prediction mode 4 from the position of these blocks in consideration of the margin value (135) The area existing at ˚) is selected as the edge area.

After the prediction of the edge direction of the lost macroblocks and the separation of the edge region and the flat region through the preprocessing process, the lost macroblocks are restored using different interpolation techniques in each region. First, in the case of the edge region, pixel values are restored through a directional interpolation technique using the predicted edge direction, and FIG. 6 is an exemplary diagram for explaining this.

In FIG. 6, P _C denotes a pixel value in an edge region, and P _i denotes an outer pixel value of a lost macroblock. The pixel value P _C in the edge region is calculated by first-order linear interpolation of two external pixel values that pass through this point and exist on a straight line in the predicted edge direction θ. Same as 2).

식 (2)

Formula (2)

In this case, when the formula (2) it calculating the predominant prediction mode is selected as the prediction mode 5 to the prediction mode 8, one of the P _i is there is point to the value of the half pixel position, a value of the If P _i is adjacent to the P _i It is used by substituting the average of the pixel values of two integer positions.

After restoring the edge region, the flat region is restored using the weighted average of four known pixel values located in the horizontal and vertical directions currently used in H.264. In addition to the pixel values, the pixel values of the edge region already restored are used for the restoration.

FIG. 7 shows the calculation amount required for the implementation of the present invention in comparison with the previous method when the edge area in the lost macroblock occupies 50%, and a large value in the calculation amount in the present invention is the dominant prediction mode. A value in mode 8 is selected, and a small value represents a case in which a prediction mode other than prediction mode 5 to prediction mode 8 is selected as the dominant prediction mode.

As shown in FIG. 7, by using the surrounding prediction mode known to predict the edge direction of the lost macroblock, the direction prediction of the edge of the macroblock and the edge region and the flat region are classified with only a small amount of computation. The method, of course, has a smaller amount of multiplication than the reference software, and the present invention may have a lower amount of calculation in view of the direction of the various edges in the image.

For the evaluation of the present invention, Akiyo, Coastgaurd, Container, Foreman, Highway, Silent sequences of CIF (352x288) size including edge components in various directions were used as test sequences, and each sequence was H.264 reference software (JM10.1). [9]) was used to encode the first 30 frames into I frames, and the H.264 baseline profile was used for the encoding process.

The attached FIG. 8 shows the objective image quality in each sequence when the macroblock loss rate of 10% occurs, and shows the results of the method currently used in the H.264 reference software and the Alkachouh method based on the fast DCT for comparison. Presented together.

At this time, the margin of the edge area was used. As can be seen in Figure 8, the error concealment method according to the present invention exhibits high objective image quality compared to the other two methods. In particular, when there are many diagonal edge components such as Foreman image, the error concealment method according to the present invention improves the objective image quality by about 3.2 to 5.5 dB compared with the other two methods.

9 shows the subjective picture quality when 20% macroblock loss occurs for the first frame of the foreman sequence. As can be seen from the results, it can be seen that the image reconstructed using the error concealment method according to the present invention has superior subjective image quality compared to the case of using the other two methods.

According to the spatial error concealment method having the low complexity in the present invention H.264 having the above-described configuration and preferred embodiment, by using the fact that the prediction mode information of the intra block is significantly related to the edge direction in the block, Predict the edge direction of the lost macroblock using the prediction mode information of the block located in the periphery of the lost macroblock in the intra frame, separate the lost macroblock into the edge region and the flat region, and then use different interpolation techniques. By using, since the lost macroblock can be easily recovered, it is possible to quickly recover the lost macroblock using only a small amount of computation, and even if the macroblock in the intra frame is lost by having only a small amount of computation. There is an advantage that can decode a good quality image in real time.

In addition, the present invention recovers an error using a prediction mode used for encoding an intra frame in the H.264 video standard when a macroblock in an intra frame is lost. Since the prediction mode is highly related to the direction of the edges in the image, it can be efficiently used for error concealment, and the computation can be minimized because the information is already included in the H.264 bitstream.

Claims (6)

In a spatial error concealment method having a low complexity in H.264, Predicting the direction of the edge of the lost macroblock by selecting a dominant prediction mode among prediction modes of neighboring blocks located around the lost macroblock; Classifying a region in the lost macroblock into an edge region in which an edge exists and a flat region in which no edge exists using the selected predominant prediction mode; The edge region is restored by the edge-based directional interpolation technique, and the flat region is restored by interpolation technique using a weighted average to restore the lost macroblock. Complex error concealment method with complexity. The method according to claim 1, The edge direction of the lost macroblock is estimated using the prediction mode for 16 4x4 blocks located on the top, bottom, left, and right sides of the lost macroblock, and the size of the edge in each block. Spatial error concealment method with low complexity. The method according to claim 2, The size of the edge of each 4x4 block selects four pixel values in which each block is perpendicular to the prediction direction of the prediction mode used in the encoding process, and the maximum and minimum values among the four selected pixel values. A spatial error concealment method with low complexity in H2.446, which is determined by the difference of. The method according to claim 3, The dominant prediction mode is determined as a prediction mode in which the sum of edge sizes in each prediction mode among the prediction modes in the sixteen 4x4 blocks is maximum, and the prediction direction of the dominant prediction mode is an edge in the lost macroblock. Spatial error concealment method with low complexity in H24, characterized by determining in the direction. The method according to claim 1, The edge region selects a block having the selected prevailing prediction mode among neighboring blocks located around the lost macroblock, and determines a region existing in the prediction direction of the selected prevailing prediction mode in the selected block. A spatial error concealment method having a low complexity in H 246.4. The method according to claim 5, And said edge region further comprises a margin of a predetermined region.

Images