KR20000067180A - Apparatus for picture restoration and error concealment method therefor - Google Patents

Abstract

PURPOSE: An image recovering apparatus and an error hiding method thereof are provided to use a weighted value according to the direct value of adjacent MB and the sub-block or the position of motion vector while recovering an image data. CONSTITUTION: An image recovering apparatus includes a decoder(300), an error detector(302), a weighted error hiding portion(304) and an image recovering portion(306). The decoder(300) decodes bit streams of received picture data on a slice basis. The error detector(302) detects an error occurred in a macro block configuring slices during decoding. The weighted error hiding portion(304) attains characteristic information of current macro block with those of neighboring macro blocks which are decoded in a way that the information is weighted based on a spatial correlation with the neighboring macro blocks so as to hide errors. The image recovering portion(306) restores image from decoded data which are generated at the weighted error hiding portion and the data decoded at the decoder.

Description

Apparatus for picture restoration and error concealment method therefor}

The present invention relates to an error-resilient image restoration apparatus and an error concealment method therefor, and more particularly, to an image restoration apparatus for effectively restoring an image by coping with an error generated during image data transmission and an error concealment method therefor. will be.

When transmitting a video signal through an error-prone communication network such as a wireless channel, the bit string received at the receiving end cannot avoid deformation and loss of some bits and consecutive bits due to a transmission error. Lost and deformed bits cause significant deterioration in the decoded image.

Therefore, the task of correcting the lost and modified bits is important, and such loss correction is an indispensable technique in transmitting video signals over a mobile network and the Internet such as the International Mobile Network (IMT) -2000.

One of the loss correction methods described above is an error concealment method. The error concealment method is to restore the information in error by using the decoded data without error and transform it into a form that does not obscure human vision.

1 shows a data structure of a conventional error concealment method. The data structure according to FIG. 1 includes a plurality of slices or groups of macro blocks 100, 102, 104, and 106 that are transmitted, and a plurality of slice memories 108, 110, 112, and 114 that store decoded slice data. And a frame memory 116 that stores data stored in the slice memories so as to reproduce the slice in a frame form without error.

The error concealment method according to FIG. 1 is as follows. First, a decoder (not shown) detects a slice start code (GBSC) from the transmitted bit stream and decodes each slice (100, 102, 104, 106). The decoded slice data is stored in the slice memories 108, 110, 112, and 114. gn represents a slice number. Data stored in the slice memories 108, 110, 112, and 114 are stored in the corresponding location of the frame memory 116 and restored. At this time, if an error is detected in the second slice 102, the decoded second slice data 110 is deleted, and the motion vectors stored in the first slice memory 108 are compensated to compensate for the frame memory 116. Save to that location. If the first slice data is not available. The motion vector is set as a zero vector and is simply copied from the immediately decoded picture.

However, the TCON method has a problem that motion compensation using the motion vector of the previous MB is impossible because there is no previous picture when the first picture is an intra-coded picture. Therefore, Intra coded pictures cannot use the TCON method.

2 illustrates a conventional method of compensating a motion vector when the previous MB is in an INTER4V mode having four motion vectors. Reference numerals 201 and 202 denote MBs of an error-free slice, respectively, and 211 and 212 denote MBs of an errored slice. 201 and 211 are characterized by a motion vector, in which the MB mainly contains information about a portion of the object which is not in motion or is not at the boundary of the object, such as the inside or the background of the object. 202 and 212 are cases in which the MB has information on the part of the object moving or the boundary of the object. In this case, the MB obtains a motion vector for four sub-blocks. As shown, the motion vector of the MB in which the error occurs is obtained by copying the motion vectors of the MB located immediately above without error.

However, if the motion vectors have strong spatial correlation, such as MB in INTER4V mode, it may be possible to directly compensate the motion vector of the MB in error by copying the motion vector of the previous MB as shown in FIG. 2. There is a problem that vertical and horizontal discontinuities occur in the reconstructed image. That is, although the upper two motion vectors of 212 have a higher similarity with the lower two motion vectors than the upper two motion vectors of 202, there is a problem of reconstructing the upper two motion vectors having lower correlation.

The technical problem to be achieved by the present invention is to reconstruct an image by compensating for a DC value or a motion vector of an MB in which an error occurs and to restore the DC value or motion vectors of sub-blocks of neighboring MBs which have been previously decoded according to the position of each subblock. The present invention provides an image restoration apparatus that compensates by giving a weight, and an error concealment method for the same.

1 shows a data structure of a conventional error concealment method.

2 shows a conventional method of compensating for a motion vector when the previous macro block is in an INTER4V mode with four motion vectors.

3 is a block diagram of an image reconstruction apparatus according to the present invention.

FIG. 4A shows one picture used in the present invention, and FIG. 4B shows one macro block.

5 shows a slice decoded without an error and a slice in which an error has occurred.

6 illustrates a slice structure including a slice in which an error occurs.

7A to 7D are performance comparison diagrams for Intra coded pictures.

According to an aspect of the present invention, there is provided a device for reconstructing an image by receiving image data including a slice consisting of a plurality of macroblocks and pictures consisting of a plurality of slices, wherein the bit of the received picture data is restored. A decoder which decodes the stream in slice units; An error detector which detects an error generated in a macroblock constituting a slice during decoding; When concealing a current macroblock in which an error is detected, the feature information of the current macroblock is obtained by combining feature information of neighboring macroblocks that have already been decoded, and according to the spatial correlation with the neighboring macroblocks. A weighted error concealment unit which conceals and combines errors by giving a predetermined weight to them; And an image reconstruction unit for reconstructing an image from the data decoded by the decoder and the decoded data generated by the weighted error concealment unit.

In order to achieve the above technical problem, the present invention conceals an error generated in a received image signal when reconstructing an image by receiving image data including a slice consisting of a plurality of macroblocks and pictures consisting of a plurality of slices. A method, comprising: decoding a bit stream of received picture data in units of slices; Detecting an error occurring in the macro block constituting the slice during decoding; And when concealing a current macroblock in which an error is detected, obtains feature information of the current macroblock by combining feature information of neighboring macroblocks that have already been decoded, according to spatial correlation with the neighboring macroblocks. And combining and concealing errors by giving a predetermined weight to the information.

According to an aspect of the present invention, there is provided a method for concealing an error generated in a received image signal when reconstructing an image by receiving image data including a slice composed of a plurality of macroblocks and pictures consisting of a plurality of slices. A method comprising: decoding a bit stream of received picture data in units of slices; Detecting an error occurring in the macro block constituting the slice during decoding; Decoding the next slice of the slice in which the error was detected; And when concealing a current macroblock in which an error is detected, obtains feature information of the current macroblock by combining feature information of neighboring macroblocks that have already been decoded, according to spatial correlation with the neighboring macroblocks. And combining and concealing errors by giving a predetermined weight to the information.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

3 is a block diagram of an image reconstruction apparatus according to the present invention. The image restoration apparatus according to FIG. 3 includes a decoder 300, an error detector 302, a weighted error concealment unit 304, and an image restoration unit 306. The operation of the image restoration apparatus according to FIG. 3 is as follows.

The decoder 300 detects the GBSC from the received video signal bitstream and decodes the slice unit.

The error detector 302 detects an error generated in the received bitstream. An error is detected when:

1) Codeword error: This is the most frequent case.

2) After decoding one slice, the same sync code (SYNC) as the next GBSC is not found.

3) The motion vector deviates from the picture even though advanced prediction methods such as INTER4V mode are not allowed.

4) When the variable length coding coefficient table is not adaptively selected, the recovered discrete cosine transform (DCT) coefficient point is out of 63 positions.

5) The chromaticity DC value is out of normal.

Cases 1) and 2) are the most important error detection criteria, and errors can be detected even in cases other than the above five cases.

When an error is detected by the error detector 302, the weighted error concealment unit 304 conceals the error by weighting the feature values of the neighbor MBs according to the importance of the neighbor MBs to generate a feature value of the MB in which the current error has occurred. do.

The image reconstructor 306 reconstructs the image from the slice data decoded by the decoder 300 and the slice data decoded because the error is concealed by the weighted error concealer 304.

Meanwhile, the weighted error concealment method according to the present invention will be described in more detail. In general, the information which has the greatest influence on the reproduced video is information indicating the encoding mode used in each MB. The encoding modes include an Intra (INTRA) mode using no motion vector, an Inter (INTER) mode using one motion vector, and an INTER4V mode using four motion vectors, provided that the quantization step has a constant size. have. Next to the encoding mode information, information having a large influence on the image quality of the reproduced image is DC information of DCT coefficients in the case of Intra MB and motion vector information in the case of Inter MB. Inter MB uses the information in the previous image due to its nature, but in the case of Intra MB, since there is no previous image, it must give high error robustness to DC data. In addition, in the case of motion information (motion vector), it is possible to recover using motion information of adjacent MBs. The AC information in the DCT coefficients except for the movement information and the DC information has a relatively small effect on the playback image. Therefore, in the present invention, all DCT AC values of the MB in which the current error occurs in the Intra coded picture are set to "0".

The error concealment method according to the present invention first extracts the DC information of each MB in the Intra mode and the motion vector information of the MBs in the Inter and INTER4V modes as described above. 4A shows one picture used in the present invention, and FIG. 4B shows one MB.

Reference numeral 400 denotes one picture, 402 denotes one slice, and 404 denotes one MB. 406 shows a sub block for MB 404 of FIG. 4A.

As shown, the (i, j) th MB is represented by M (i, j). M (i, j) is four subblocks representing the luminance component of MB, b _i (i, j) (i = 1,2,3,4) and two subblocks representing the chromaticity components C _r , C _b , b _i (i, j) (i = 5,6), which can be expressed as

FIG. 5 illustrates slices decoded without errors and slices with errors for explanation of the present invention. 510 denotes a slice (l-1) decoded without an error, and 520 denotes a slice (l) in which an error occurs, and it is assumed that the MB 521 of the current hatched portion is restored.

First, the Intra coded picture will be considered. For example, the DC value of the upper MB 512 is DC ₁ (k, l-1) ≠ DC ₃ (k, l-1) or DC ₂ (k, l-1) ≠ DC ₄ (depending on the subblock). In the case of k, l-1), if the DC value of the current MB is copied as DC _i (k, l) = DC _i (k, l-1), the reconstructed image will appear discontinuously. Accordingly, in the present invention, the DC value of the current MB, DC _i (k, l), is obtained from the DC value of neighbor MBs 511, 512, 513 in consideration of the vertical correlation of the image data as follows.

As another embodiment of error concealment in the Intra coded picture, DC _i (k, l) may be obtained by considering the horizontal correlation when obtaining the DC value of the current MB as follows.

Here, w ₁ , w ₂ , w ₃ are weights having the characteristics of w ₂ > w ₁ , w ₃ , respectively, and m is a value added to compensate for integer truncated error, and used in the present invention. Value is +2.

In consideration of the vertical correlation, the DC value of the subblocks having the indexes 3 and 4 is the same as the DC value of the subblocks having the indexes 1,2.

As another embodiment of error concealment in an intra coded picture, a DC value of a subblock having an index of 1,2 is obtained in the same manner as in Equation 3 above, and a DC value of a subblock having an index of 3,4 is of l + 1th. Consider the DC values of the corresponding subblocks of the slice. 6 shows a slice structure thereof. In this case, the restoration of the image is performed by delaying one slice. For example, after decoding the l-1 th slice 610 as shown, after decoding the l th slice 620, and after the l th slice 620 is completely decoded, 610 is restored to the image. After decoding the l + 1 th slice 630 and the l + 1 th slice 630 is completely decoded, the l th slice 620 is restored. Here, if it is determined that there is an error in the l-th slice 620 (1) while decoding the l-th slice 620, the decoding of the l + 1 th slice 630 is performed, and the l + 1 th slice 630 is decoded. After the decoding is completed, the error of the first slice 620 is concealed using the decoded data of the l + 1 th slice 630 and the decoded data of the l−1 th slice 610. The DC value of the current MB according to this embodiment can be obtained as follows.

In case of concealing an error of an inter coding picture, it is divided into a motion vector recovery (Motion Vetor Reconstruction) step and an MB recovery (Macro Block Reconstruction) step. Each step will be described with reference to the slice structure shown in FIG. 5. The steps for recovering the motion vector are divided as follows.

1) If the current MB 521 is in INTER mode and the upper MB 512 is in INTER mode: v ₀ (k, l) = v _o (k, l-1)

2) If the current MB 521 is in INTER4V mode and the upper MB 512 is in INTER mode: v _i (k, l) = v _o (k, l-1), i = 1,2,3,4

3) When the current MB 521 is in the INTER4V mode and the upper MB 512 is in the INTER4V mode, it is determined by considering the vertical correlation as follows.

At this time, v _i = [2 2] ^T may be used to compensate for integer truncation errors.

In the case of 3) described above, considering the vertical and horizontal correlations, the motion vector of the current MB can be obtained as follows.

The steps for recovering the MB are as follows. When the upper MB 512 is in the INTER mode, a 16x16 block motion compensation method is used, and when the upper MB 512 is in the INTER4V mode, an overlapped block motion composition (OBMC) method is used. Examples of weights for OBMC are shown in the following table.

Weight applied to the motion vector of the current subblock 4 5 5 5 5 5 5 4 5 5 5 5 5 5 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 5 5 5 5 5 5 4

Weights applied to the upper or lower subblocks of the current subblock 2 2 2 2 2 2 2 2 One One 2 2 2 2 One One One One One One One One One One One One One One One One One One One One One One One One One One One One One One One One One One One One 2 2 2 2 One One 2 2 2 2 2 2 2 2

Weight applied to left or right subblock of current subblock 2 One One One One One One 2 2 2 One One One One 2 2 2 2 One One One One 2 2 2 2 One One One One 2 2 2 2 One One One One 2 2 2 2 One One One One 2 2 2 2 One One One One 2 2 2 One One One One One One 2

In this case, the weights are calculated by Orchard and Sullivan (OBMC: an estimation-theoretic approach, IEEE trans.on image processing, vol. 3, No. 5, pp693 ~ 699). , Sept. 1994), by the Lagrange multiplier method.

The following tables show the conditions and results used in the simulations for the present invention.

Characteristics of Experimental Pictures Used in Simulation Test sequence format Frame rate (frams / sec) Channel bitrate (kbps) Code rate (kbps) Container QCIF 10 64 48 Foreman QCIF 7.5 64 48 Hall QCIF 10 64 48 News QCIF 10 64 48

Channel Error Patterns Used in the Simulation Channel Error Pattern Doppler Frequency (Hz) Average bit error rate Average burst length (bits) E1 5.3 1.35 x 10 ^-5 16 E2 70 1.26 x 10 ^-3 17 E3 211 9.73 x 10 ^-4 15 E4 5.3 8.17 x 10 ^-5 11 E5 70 1.21 x 10 ^-4 13 E6 211 9.37 x 10 ^-5 11

The first method is the method set the DCT AC value to "0", the second and the third method if each _{w 1 = w 3 = 0,} w 2 = 1 in Equation 3, w ₁ = w ₃ = 1 and w ₂ = 2.

The following is a simulation result according to the error concealment method of an inter coded picture. At this time, the Intra coded picture was concealed by the above-described third method.

Comparison of Peak Signal-to-Noise Ratio (SNR) for Inter Coded Pictures Container Foreman Hall News error Conventional The present invention Conventional The present invention Conventional The present invention Conventional The present invention E1 21.78 21.87 18.00 18.05 23.28 23.31 20.61 20.65 E2 24.57 30.75 16.59 16.63 26.36 26.40 22.41 22.45 E3 23.42 23.50 17.56 17.60 24.26 24.28 22.75 22.78 E4 29.98 30.04 28.56 28.60 34.02 34.04 31.29 31.33 E5 31.63 31.67 29.16 29.19 33.07 33.09 30.30 30.32 E6 30.70 24.69 28.29 28.32 33.89 33.89 29.22 29.22 Average 27.01 27.09 23.03 23.07 29.15 29.17 26.10 26.13

Here, the method of the present invention is the case where w ₁ = w ₃ = 0 and w ₂ = 1 in Equation 6.

As shown in the tables, it can be seen that the peak SNR by the error concealment method of the present invention is improved compared to the conventional method.

7A to 7D are performance comparison diagrams for Intra coded pictures.

The test sequence is 'Foreman' and the error pattern of E3 is used. 7A shows the result of restoration without error concealment. 7B to 7D illustrate a method in which the DCT AC value is set to "0", a method in which w ₁ = w ₃ = 0 and w ₂ = 1 in Equation 3, w ₁ = w ₃ = 1, w ₂ This is the result of setting the method to = 2. In the case of FIG. 7B, the black band portion is reduced compared to FIG. 7A, which is restored without error concealment, but there are still discontinuous blocking in the vertical and horizontal directions. In the case shown in FIG. 7B, no vertical discontinuity pattern is generated, but a horizontal discontinuity pattern still remains. The case shown in Figure 7d shows superior performance compared to other methods.

According to the present invention, since syntaxes of neighboring MBs without errors are combined in consideration of horizontal and vertical correlations, and the combined values are replaced with syntaxes of MBs with errors, discontinuous patterns that may occur in the horizontal and vertical directions are generated. Can be reduced.

Claims (12)

An image restoration apparatus for reconstructing an image by receiving image data including a slice composed of a plurality of macro blocks and pictures composed of a plurality of slices, A decoder which decodes the bit stream of the received picture data in units of slices; An error detector which detects an error generated in a macroblock constituting a slice during decoding; When concealing a current macroblock in which an error is detected, the feature information of the current macroblock is obtained by combining feature information of neighboring macroblocks that have already been decoded, and according to the spatial correlation with the neighboring macroblocks. A weighted error concealment unit which conceals and combines errors by giving a predetermined weight to them; And And an image restoring unit for restoring an image from the data decoded by the decoding unit and the decoded data generated by the weighted error concealment unit. The method of claim 1, wherein the weighted error concealment unit When each macro block is divided into four sub blocks according to a luminance component, the feature information of the upper sub blocks of each sub block of the current macro block is determined by weighting the feature information of the lower sub blocks of the neighboring macro blocks. And feature information of the lower subblocks of the current macroblock is replaced with feature information of the determined upper subblocks of the current macroblock. A method of concealing an error occurring in a received image signal when reconstructing an image by receiving image data including a slice consisting of a plurality of macro blocks and pictures consisting of a plurality of slices, Decoding the bit stream of the received picture data in units of slices; Detecting an error occurring in the macro block constituting the slice during decoding; And When concealing a current macroblock in which an error is detected, the feature information of the current macroblock is obtained by combining feature information of neighboring macroblocks that have already been decoded, and according to the spatial correlation with the neighboring macroblocks. And concealing the error by combining the predetermined weights. The method of claim 3, wherein the current macro block information is If the picture is Intra coded, the slice decoded immediately before the slice is referred to as the l-1th slice starting from the upper left, the position of the current macroblock is called the kth macroblock, and each macroblock is represented by luminance. When dividing into four subblocks according to the component, the DC value DC _i (k, l) (i = 1,2,3,4) of the discrete cosine variation coefficient of each subblock with respect to the current macroblock is determined. , DC _i (k, l) is given by the vertical correlation (Equation 2) Error concealment method when restoring an image, characterized in that as follows. The method of claim 3, wherein the current macro block information is If the picture is Intra coded, the slice decoded immediately before the slice is referred to as the l-1th slice starting from the upper left, the position of the current macroblock is called the kth macroblock, and each macroblock is represented by luminance. When dividing into four subblocks according to the component, the DC value DC _i (k, l) (i = 1,2,3,4) of the discrete cosine variation coefficient of each subblock with respect to the current macroblock is determined. , DC _i (k, l) is the following equation according to the vertical and horizontal correlation Here, w ₁ , w ₂ , w ₃ are weights each having the characteristics of w ₂ > w ₁ , w ₃ , and m is a value added to compensate for integer truncation error. Error concealment method when restoring an image, characterized in that as follows. The method of claim 5, When dividing the current macroblock into two blocks for chromatic components, the DC values DC _i (k, l) (i = 5,6) of the two blocks are Error concealment method during image restoration, characterized in that determined as follows. The method of claim 3, wherein the current macro block information is When the picture is inter-coded, the slice immediately decoded before the slice is called the l-1th slice starting from the upper left corner, and the position of the current macroblock is called the kth macroblock. When divided into four subblocks according to the component, the motion vectors v _i (k, l) and i = 1,2,3,4 of each subblock for the current macroblock are determined. When the current macroblock is in the INTER mode and the upper macroblock of the current macroblock is in the INTER mode, the motion vector is v ₀ (k, l) = v _o (k, l-1) (v ₀ is a macroblock. Indicating a motion vector in the INTER mode); If the current macroblock is an INTER4V mode and the upper macroblock of the current macroblock is an INTER mode, the motion vector is v _i (k, l) = v _o (k, l-1), i = 1,2, Determining 3,4 (v ₀ represents a motion vector when the macro block is in the INTER mode); And When the current macroblock is INTER4V mode and the upper macroblock of the current macroblock is INTER4V mode, (Equation 5) Error concealment method when restoring an image, characterized in that it comprises the step of determining. The method of claim 7, wherein the determining of the motion vector according to the vertical correlation The motion vector of the current macro block according to the vertical and horizontal correlation (Equation 6) Error concealment method when restoring an image, characterized in that it comprises the step of determining. The method of claim 7, wherein when the motion vector is determined and then compensated for using the motion vector, If the upper macro block is in the INTER mode, compensating with a 16 × 16 block motion compensation method; And If the upper macroblock is an INTER4V mode, the block is compensated using an overlapping block motion compensation (OBMC) method. and determining according to the Lagrangian multiplication method according to a vertical or horizontal correlation with a neighboring macroblock located at (k + 1, l). A method of concealing an error occurring in a received image signal when reconstructing an image by receiving image data including a slice consisting of a plurality of macro blocks and pictures consisting of a plurality of slices, Decoding the bit stream of the received picture data in units of slices; Detecting an error occurring in the macro block constituting the slice during decoding; Decoding the next slice of the slice in which the error was detected; And When concealing a current macroblock in which an error is detected, the feature information of the current macroblock is obtained by combining feature information of neighboring macroblocks that have already been decoded, and according to the spatial correlation with the neighboring macroblocks. And concealing the error by combining the predetermined weights. The method of claim 10, wherein the characteristic information of the current macro block is If the picture is intra coded, the slice just decoded immediately before the slice is the l-1th slice from the top left, the next slice is the l + 1th slice, and the position of the current macroblock is the kth macro. When the macroblock is divided into four subblocks according to luminance components, the DC value of the discrete cosine variation coefficient of each subblock with respect to the current macroblock DC _i (k, l) (i = 1, 2,3,4), and DC _i (k, l) is given by the following equation according to the vertical and horizontal correlation Here, w ₁ , w ₂ , w ₃ are weights each having the characteristics of w ₂ > w ₁ , w ₃ , and m is a value added to compensate for integer truncation error. Error concealment method when restoring an image characterized in that the value is determined as. The method of claim 11, When dividing the current macroblock into two blocks for chromatic components, the DC values DC _i (k, l) (i = 5,6) of the two blocks are Error concealment method during image restoration, characterized in that determined as follows.