JP4570081B2 - Moving image error concealment method and apparatus - Google Patents

Description

  TECHNICAL FIELD The present invention relates to a moving image error concealment method and apparatus, and more particularly to a moving image error concealment in which reproduction image quality is prevented from abruptly changing during decoding of encoded data to reduce subjective image quality degradation. It relates to a method and an apparatus.

  As a conventional technique for reducing the deterioration of the image quality of a decoded image of an encoded moving image, for example, there is one disclosed in Patent Document 1 below. This patent document 1 discloses the following technique. For example, when an inter-image data portion and an intra-image data portion are mixed in an image signal input on one raster, a high frequency component is generated at the changed portion. For this reason, when the filtering output for the intra portion is quantized with the same quantization characteristics as the filtering output for the inter portion, a large quantization distortion occurs, and the image quality of the decoded image is deteriorated. Therefore, according to the present invention, when non-block orthogonal transform processing such as wavelet transform is performed by performing filtering processing so as not to include high frequencies from the periphery of the intra region to the intra region, The generation of high-frequency components due to such changes is reduced, and deterioration of output data after decoding is avoided.

Another conventional technique is described in Patent Document 2 below. The invention described in Patent Document 2 is as follows. For example, in a system with a low encoding rate, when an image with a large amount of motion or an image that has undergone a scene change is decoded, encoding noise such as block distortion is generated, and degradation of the decoded image is conspicuous. Therefore, in the decoding apparatus, the received encoded data is subjected to variable length decoding to calculate the number of intra macroblocks generated, the characteristics of the low-pass filter are determined based on the number of occurrences, and the decoded image signal is filtered. As a result, degradation of image quality is reduced.
JP-A-8-288856 Japanese Patent Laid-Open No. 10-56646

  The above-described prior art in Patent Document 1 conceals image quality degradation when an inter image data portion and an intra image data portion occur in a single image. In a system with a low conversion rate, it is proposed to conceal image quality degradation of images that are moving rapidly or that have undergone a scene change, but image quality degradation due to changes in image quality when viewed spatially or temporally There was a problem that no consideration was given to.

  For example, an encoded image of a certain frame generates an error due to a transmission failure or the like, and an error occurs in a part 50a (shaded portion) of the decoded image 50 as shown in FIG. Suppose that the error concealment process is performed. Then, the influence of this error concealment process is propagated to the image of the subsequent frame, and an error propagation area 51a diffused in the decoded image 51 is generated, for example, as shown in FIG. In order to reduce the increase in image quality degradation due to error propagation, intra-frame macroblocks are intra-encoded into the image 52 of the subsequent frame as shown in FIG. 5C, and intra refresh is performed in the error propagation area 52a. When the area 52b is generated, there is no error in the intra refresh area 52b. Next, when the decoded / reproduced image comes to the beginning of a GOP (group of pictures), an intra-picture image 53 arrives as shown in FIG. Since this intra picture 53 is an image that does not refer to other images, it is a high-quality image.

  In the above case, the error area suppresses the degradation of the image quality by the error concealment process, but the boundary between the error areas 50a, 51a, 52a and the non-error area (including the intra refresh area 52b) is the difference in the image quality. Therefore, this difference in spatial image quality subjectively degrades the image quality. Also, if the image of FIG. 7 (c) suddenly changes to the intra picture image of FIG. 7 (d), it will suddenly change from an image that is blurred due to an error to an error-free image, that is, an image without quality degradation. An abrupt temporal change subjectively results in degradation of image quality.

  In other words, the prior art has made various proposals for concealing the error image, but there is a special countermeasure for the degradation of image quality due to a sudden change from an image that is blurred due to an error to an image that does not have an error. There was a problem that no consideration was given.

  The present invention has been made in view of the above-described problems of the prior art, and an object of the present invention is to provide a moving image error concealment method capable of reducing image quality degradation due to a change in image quality when viewed spatially and / or temporally. And providing an apparatus.

In order to achieve the above object, the present invention specifies an error block based on block decoding information obtained by variable length decoding , performs error concealment processing on the error block, and converts it into a block that is not an error block. On the other hand, a higher reliability is given to the error block, and the filter strength of the block is determined so that the filter strength decreases as the reliability increases , and at least one adjacent block in space and time is determined. If the difference in filter strength between them is equal to or greater than a predetermined threshold value, the filter strength difference is corrected to a magnitude that is less than the threshold value, and the error-concealed decoded reproduced image is converted into the filter The first feature is that filtering is performed by intensity.

In addition, a variable length decoding unit that performs variable length decoding of the encoded moving image data, an error block specifying unit that specifies an error block based on block decoding information from the variable length decoding unit, and an error in the error block specifying unit an error concealment unit that performs error concealment processing to the identified block areas, a block not said error block, the reference-source block for best reliability in or non-intra blocks for intra blocks And determining the filter strength of the block so that the filter strength decreases as the reliability increases , based on the reliability determined by the reliability determination unit. If the difference in filter strength between at least one adjacent block is equal to or greater than a predetermined threshold, the difference in filter strength is less than or equal to the threshold. A filter strength determining unit for correcting the filter strength, and a filter applying unit for filtering the decoded reproduction image subjected to the error concealment processing with the filter strength determined by the filter strength determining unit. There are features.

  According to the present invention, the filter strength is obtained based on the reliability of the block, and when the difference in filter strength between at least one spatial and temporal adjacent blocks is equal to or greater than a predetermined threshold, the filter strength is determined. The filter strength is modified so that the difference is less than or equal to the threshold value, and the decoded reproduced image is filtered with the filter strength, so that the recovery of error blocks over time becomes gradual, and subjective image quality is degraded. It can be greatly reduced.

  In addition, when the filter strength difference between spatially adjacent blocks is equal to or greater than a predetermined threshold, the filter strength is corrected so that the filter strength difference is equal to or smaller than the threshold. Error block recovery becomes gradual, and subjective image quality degradation can be greatly reduced.

  In addition, even when error concealment is properly performed in the error area and the error is not perceived very subjectively, the image quality will change drastically due to the appearance of the intra picture, and the error will occur before that. It is also possible to solve the problem that the observer will later recognize that there has been.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a decoder including the present invention.

  MPEG or H.264 The moving image data encoded by 26X or the like is input to the variable length decoding unit 1. The variable length decoding unit 1 performs variable length decoding on the input moving image data a, and restores an encoding mode, a motion vector, a quantized orthogonal transform coefficient, and the like for each macroblock of the image.

  Information indicating whether the encoding mode and the decoding in units of macroblocks have been appropriately performed or whether the decoding has not been properly performed due to an error or the like is sent to the error block specifying unit 2 as block decoding information b. The motion vector information c is sent to the motion compensation unit 3. The quantized orthogonal transform coefficient d is sent to the inverse quantization unit 4.

  The inverse quantization unit 4 performs inverse encoding of the quantized orthogonal transform coefficient d to obtain an orthogonal transform coefficient e. The orthogonal transformation coefficient e is inversely orthogonal transformed by the inverse orthogonal transformation unit 5 to become image information f.

  The motion compensation unit 3 extracts the area of the image from the frame memory 6 based on the motion vector information c sent from the variable length decoding unit 1. The extracted image of the area is added to the image information f by the adding means 7 to generate a reproduction image.

  Based on the block decoding information b, the error block specifying unit 2 specifies an area that has not been correctly decoded in units of macroblocks. An error concealment unit 8 performs an error image concealment process, for example, on a frame basis for an image including a macroblock that is determined to have not been correctly decoded by the error block identification unit 2. This error concealment process can be performed by a conventionally known method. For example, it can be performed by a filter process as disclosed in Patent Documents 1 and 2 and the like.

  The reproduced image g is generated by an image obtained by the motion compensation, an image f obtained by inverse orthogonal transformation, and an error concealment image h generated by the error concealment unit 8 when there is an error. The reproduced image g is stored in the frame memory 6 and sent to the filter application unit 11 to be used as a reference for encoding subsequent images.

  The reliability determination unit 9 determines whether or not decoding without error is performed on a macroblock basis, and determines the reliability based on the determination result. The filter strength determination unit 10 determines the filter strength to be applied to the reproduced image g based on the reliability in units of macroblocks obtained by the reliability determination unit 9. The filter application unit 11 applies a filter with the filter strength determined by the filter strength determination unit 10 to the reproduced image g. The image j to which the filter is applied is output for display or the like.

  Next, functions of the error block specifying unit 2, the reliability determining unit 9, and the filter strength determining unit 10, which are main parts of the present invention, will be described with reference to the flowcharts of FIGS. 2, 3, and 5, respectively. .

  The error block specifying unit 2 determines whether or not there is a block decoding error based on the block decoding information b in step S1 of FIG. If this determination is negative, the process proceeds to step S5 to determine whether or not the last macroblock (hereinafter referred to as MB) of one frame has been reached. When this determination is negative, the process proceeds to step S6 to update the MB.

  If the determination in step S1 is affirmative, that is, if it is determined that there is a decoding error in the MB, the process proceeds to step S2, and the starting point of the MB that cannot be correctly decoded is specified. Next, the process proceeds to step S3, and an MB that can be decoded again is detected. In step S4, the end point of the MB that cannot be decoded is specified. When the above operation is repeated and the determination in step S5 becomes affirmative, the series of processes described above ends.

  As is clear from the above, the error block specifying unit 2 specifies from the start point to the end point of the MB that cannot be correctly decoded.

  Next, the reliability determination unit 9 determines whether or not the MB is an error block in step S11 of FIG. 3 based on the information from the error block specifying unit 2. If this determination is affirmative, the process proceeds to step S12 to lower the block reliability. For example, if the reliability of an MB without error is “10”, the reliability is lowered to 9 or less.

  On the other hand, when the determination in step S11 is negative, the process proceeds to step S13 to determine whether or not the MB is an intra block. If this determination is affirmative, the process proceeds to step S14 to maximize the block reliability. For example, the reliability is set to “10”. If the determination in step S13 is negative, the process proceeds to step S15 to specify the encoding reference source. For example, in the case of a P picture, it is specified whether it is an I (intra) picture or another P picture. In the case of a B picture, an I or P picture that is before and after in time is specified.

  In step S16, the reliability of the reference source block specified in step S15 is copied. In step S17, the block reliability of all MBs in the frame is determined based on the results of steps S12, S14, and S16.

  A specific example of the operation of FIG. 3 will be described with reference to FIG. FIG. 4 shows a case where the reliability level is 10 levels from a minimum of 1 to a maximum of 10. Note that the hatched macroblock in the figure is the target MB.

  As shown in FIG. 4 (a), when the target MB is intra-coded without using inter-frame or intra-frame prediction, the reliability of the target MB is set to 10 (the step S14). ).

  As shown in FIG. 6B, in the MB in which the target MB uses intraframe prediction coding, the region used for prediction is specified, and the reliability of the current MB is determined based on the reliability information of the MB. To do. In determining the reliability of the current MB based on the reference area, it is possible to use the reliability of the MB including the most reference area, or a weighted average based on the ratio of the area (step S16).

  As shown in FIG. 5C, in the MB in which the target MB uses inter-frame predictive coding in a P picture, an area used for prediction is specified, and the current MB is determined based on reliability information of the MB. The reliability is set (step S16). In determining the reliability of the current MB based on the reference area, it is possible to use the reliability of the MB including the most reference area, or a weighted average based on the ratio of the areas.

  Furthermore, as shown in FIG. 6 (d), in the case where the target MB uses inter-frame predictive coding in a B picture, two regions used for prediction are specified, and the reliability of the MB The reliability of the current MB is determined based on the degree information (step S16). In the determination of the reliability of the current MB based on the reference area, in the prediction in each direction, the average value of the reliability of the MB including the most reference area is used, the weighted average based on the ratio of the reference areas in both directions Is available.

  Next, the operation of the filter strength determination unit 10 will be described with reference to FIG. Based on the block reliability from the reliability determination unit 9, the filter strength determination unit 10 confirms the reliability of the target MB in step S21. In step S22, the filter strength is determined to be smaller as it is inversely proportional to the reliability or as the reliability increases. Next, in step S23, it is determined whether or not the filter strength level difference with the spatially adjacent block is equal to or greater than a threshold value. If this determination is affirmative, the process proceeds to step S24, and the filter strength is forcibly set to a magnitude such that the filter strength level difference is less than a predetermined threshold.

  When the determination in step S23 is negative or after the processing in step S24 is performed, the process proceeds to step S25 to determine whether or not the difference in filter strength level with respect to temporally adjacent blocks is equal to or greater than a threshold value. If this determination is affirmative, the process proceeds to step S26, and the filter strength is forcibly set to a magnitude such that the filter strength level difference is less than a predetermined threshold. When the determination in step S25 is negative, or after the process of step S26 is performed, the process proceeds to step S27. In step S27, the filter strength is determined.

  The processing in steps S23 to S26 is to keep the difference in image quality between the spatially and temporally adjacent MBs below a certain level. In addition, although it is preferable embodiment to provide said step S23-S26, this invention is not limited to this, Either one of step S23 and S24 or step S25 and S26 may be sufficient.

For example, the filter strength determination unit 10 can set as follows when the filter strength is set to 10 levels.

Filter strength = Maximum reliability-Reliability (1)

Or

Filter strength = Maximum reliability / Reliability

  FIG. 6 is an explanatory diagram of a specific example in which the filter strength is determined from the reliability, and the filter strength is determined by the above calculation formula (1) from the reliability in MB units (left diagram). However, when the filter strength difference between MBs adjacent in time and space is compared between adjacent MBs, and the filter strength difference has an absolute difference equal to or greater than a certain value (threshold), the absolute difference is reduced. Correction is made to the filter strength. In the example shown in the figure, the filter strength difference between the target MB (shaded portion) and the right adjacent MB is “4”, and thus, for example, the threshold value is “3” or more. Therefore, the filter strength of the target MB is corrected to “2” in order to reduce the filter strength difference to be less than the threshold (right diagram). The filter strength can be determined in units of pictures or pixels in addition to setting in units of MB.

  Next, the filter application unit 11 (see FIG. 1) applies a filter to the decoded and reproduced image. Specifically, the present invention is applied to a space 3 × 3 pixels or a space 5 × 5 pixel region in units of pixels.

It is a block diagram which shows the structure of the decoder to which this invention is applied. It is a flowchart which shows the operation example of the error block specific | specification part of FIG. It is a flowchart which shows the operation example of the reliability determination part of FIG. It is explanatory drawing of a specific example of operation | movement of a reliability determination part. It is a flowchart which shows the operation example of the filter strength determination part of FIG. It is explanatory drawing of a specific example of operation | movement of a filter strength determination part. It is explanatory drawing that the image quality difference which changes rapidly in time will deteriorate image quality subjectively.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Variable length decoding part, 2 ... Error block specific | specification part, 3 ... Motion compensation part, 4 ... Inverse quantization part, 5 ... Inverse orthogonal transformation part, 8 ... Error concealment Part, 9 ... reliability determination part, 10 ... filter strength determination part, 11 ... filter application part.

Claims (5)

Identify error blocks based on block decoding information obtained by variable length decoding,
For error blocks, perform error concealment processing,
For blocks that are not error blocks, give higher confidence to error blocks,
When the filter strength of the block is determined so that the filter strength decreases as the reliability increases, and when the difference in filter strength between at least one adjacent block in space and time is equal to or greater than a predetermined threshold, Correct the filter strength so that the filter strength difference is less than the threshold,
A moving image error concealment method comprising filtering the decoded reproduction image subjected to the error concealment process with the filter strength. The moving image error concealment method according to claim 1,
The reliability of the block is the same as the reliability of the reference block for a block that is not an error block, and is lower than the reliability of a block that is not the error block for an error block. Concealment method. A variable length decoding unit for variable length decoding encoded video data;
An error block specifying unit for specifying an error block based on block decoding information from the variable length decoding unit;
An error concealment unit that performs an error concealment process on an area identified as an error block by the error block identification unit;
A block not said error block, and determines the reliability determination unit to the reliability of the referencing block for the highest confidence in or non-intra blocks for intra blocks,
Based on the reliability determined by the reliability determination unit, the filter strength of the block is calculated so that the filter strength decreases as the reliability increases , and the filter strength between at least one spatial and temporal adjacent blocks A filter strength determination unit that corrects the filter strength so that the difference in filter strength is less than or equal to the threshold when the difference is greater than or equal to a predetermined threshold;
A moving image error concealment apparatus comprising: a filter application unit configured to filter the decoded reproduction image subjected to the error concealment process with a filter strength determined by the filter strength determination unit. The moving image error concealment device according to claim 3,
The reliability determination unit sets the same reliability as that of a reference source block for a block that is not an error block, and lower than the reliability of a block that is not the error block for an error block. Concealment device. The moving image error concealment device according to claim 4,
The encoded moving image data is MPEG or H.264. In the case of moving image data encoded by 26X, the reliability determination unit maximizes the reliability of an intra-coded block.

Images