JP2012019462A - Picture decoding device and picture decoding method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that in decoding processing of moving picture coded data, a large quantity of data required to read reference data for motion compensation requires a decoding device to have high-performance memory.SOLUTION: An picture decoding device 100 decodes the coded data obtained by coding a moving picture, by decoding processing with the motion compensation. The picture decoding device 100 comprises: picture memory 112 for storing decoded picture data; a reference image acquisition unit 109 for acquiring from the picture memory 112 a reference picture required for the motion compensation; a motion compensation unit 108 for performing the motion compensation by use of the reference picture; an picture output unit 113 for reading an output picture from the picture memory 112 and outputting thereof; and a control unit 110 which determines whether writing, into the picture memory 112, of the picture data obtained by decoding the picture to be decoded will be completed within a preset target processing period, and which when determined to be not completed, performs control to interrupt decoding processing of the picture to be decoded and to restart the decoding processing from the next picture.

Description

  The present invention relates to an image decoding apparatus and an image decoding method for performing decoding processing on data obtained by compression-encoding moving images.

  In recent years, digitalization of AV information has progressed, and devices capable of digitizing and handling moving images are becoming widespread. Since a moving image has an enormous amount of information, encoding is generally performed while reducing the amount of information in consideration of recording capacity and transmission efficiency. As a moving image encoding technique, MPEG2 or H.264 is used. H.264 (see Non-Patent Document 1) has been established and is widely used in various fields such as video camera recorders, digital broadcasting, and Internet streaming. In these image coding techniques, a motion prediction (hereinafter also referred to as motion compensation) method is introduced.

  Motion prediction is processing for detecting a motion vector from a picture to be encoded and a reference picture and generating a predicted picture. Here, the reference picture is a picture that is referred to in motion prediction, and is selected from pictures that have already been encoded.

  Information such as motion information (information specifying a reference picture, motion vector, etc.) obtained by motion prediction and a difference from a prediction result is encoded. A picture is divided into rectangular areas having a size of 16 × 16 pixels, usually called macroblocks, and encoding processing is performed in units of the macroblocks.

  A typical overall configuration example of the moving image decoding apparatus for the decoding processing of the encoded image data accompanied with motion prediction as described above is shown in FIG. 3 and the operation will be briefly described. The moving image decoding apparatus includes an encoded data input terminal 101, a variable length decoding unit 102, an inverse quantization unit 103, an inverse orthogonal transform unit 104, a motion compensation unit 108, a reference image acquisition unit 109, an image memory 112, and an adder 105. And a playback image output terminal 114.

  The input image encoded data is subjected to variable length decoding by the variable length decoding unit 102, and quantized transform coefficient data and motion information are output for each macroblock of 16 × 16 pixel size. Thereafter, the decoding process is performed in units of macroblocks. The quantized transform coefficient data is subjected to inverse quantization processing by the inverse quantization unit 103 and inverse orthogonal transform processing by the inverse orthogonal transform unit 104, and prediction difference data is output. The reference image acquisition unit 109 reads out a partial area of the required reference picture from the image memory 112 based on the motion information and outputs it. The motion compensation unit 108 generates and outputs motion prediction data from the reference picture data obtained from the reference image acquisition unit 109 and the motion information. The adder 105 generates decoded pixel data by adding the prediction difference data output from the inverse transform processing unit 104 to the motion prediction data. The decoded pixel data is output from the output terminal 114 and stored in the image memory 112.

  Here, motion compensation will be described in detail. H. In H.264, a motion compensation block is 16 × 16, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, 4 × 4 pixels, compared to a macroblock composed of 16 × 16 pixels. An appropriate block size is selected from a total of seven types of motion compensation block sizes and used for encoding. A maximum of two reference pictures for generating motion prediction data can be selected from the front (temporarily past image) and the rear (temporally future image).

  FIG. 2 shows the block size of motion compensation in H.264, and shows how to divide a macroblock having a size of horizontal 16 pixels × vertical 16 pixels. First, there are four division methods for a block of 16 × 16 pixels, and each block division type is shown in FIGS.

  FIG. 4A shows a case where motion compensation is performed without dividing a macroblock (division type 1), and FIG. 4B shows that the macroblock is divided into two blocks of 16 horizontal pixels × 8 vertical pixels. FIG. 4C shows a case where motion compensation is performed by dividing a macroblock into two blocks of 8 horizontal pixels × 16 vertical pixels (division type 3). FIG. 4D shows a case where motion compensation is performed by dividing a macroblock into four blocks of 8 horizontal pixels × 8 vertical pixels (division type 4).

  Furthermore, when the division type 4 division method shown in FIG. 4D is selected, four sub-blocks of 8 horizontal pixels × 8 vertical pixels are shown in FIGS. 4E to 4H, respectively. It can be selected from a method of dividing each of 4 blocks of 8 horizontal pixels × 8 vertical pixels. Here, the division methods shown in FIGS. 4E, 4F, 4G, and 4H correspond to division type 4, division type 5, division type 6, and division type 7, respectively.

  H. In the H.264 standard, motion compensation is permitted in units of decimal pixels up to 1/4 pixel. At this time, H.C. In the H.264 standard, a 6-tap filter is employed as the linear filter pixel interpolation method, and it is determined that half-precision pixels are obtained from the surrounding six pixels. A pixel interpolation method using this 6-tap filter will be described with reference to FIG.

  FIG. 2 is a conceptual diagram for explaining a pixel interpolation method of a luminance signal in H.264. In FIG. 5, pixel signals at integer precision pixel positions before interpolation are represented by shaded squares, and pixel signals at decimal pixel positions with 1/2 pixel precision and 1/4 pixel precision are represented by white squares. First, a pixel signal with 1/2 pixel accuracy is generated from the integer pixel signal using a 6-tap filter, and then a pixel signal with 1/4 pixel accuracy is generated by an average value filter with 2 taps.

  The 1/2 pixel signal at the intermediate position in the horizontal direction between the two integer pixel signals is generated by horizontal 6-tap filter processing. For example, the pixel b is generated by applying a horizontal 6-tap filter to the integer pixels E, F, G, H, I, and J.

  A ½ pixel signal at an intermediate position in the vertical direction between two integer pixel signals is generated by vertical 6-tap filter processing. For example, the pixel h is generated by applying a vertical 6-tap filter to the integer pixels A, C, G, M, R, and T.

  The half pixel signal at the middle position of the four integer pixel signals is generated by performing 6-tap filter processing in both the horizontal and vertical directions. For example, when generating the pixel j, the signals aa, bb, b, s, gg, and hh with 1/2 pixel accuracy are generated by the horizontal 6-tap filter process, and then these signals are subjected to the 6-tap filter process in the vertical direction. Generate by doing Alternatively, after generating the pixels cc, dd, h, m, ee, and ff by the vertical filter process, j may be generated by the horizontal filter process.

  In this way, after calculating all the signals with ½ pixel accuracy, a signal with ¼ pixel accuracy is generated using the average value filter. For example, the pixel a is generated from G and b, the pixel f is generated from b and j, and the pixel r is generated from m and s by applying an average value filter.

  Therefore, in order to obtain the pixel signal at the decimal pixel position surrounded by the pixels G, H, M and N with integer pixel accuracy, the surrounding 6 × 6 pixels shown in FIG. 5 are required.

  In addition, in the block unit for performing motion compensation, as shown in FIG. 6, with respect to the area (a) where the pixels of the block for which motion compensation is performed are located, the target block is 2 pixels above and left, and 3 pixels below and right The pixels in the area (b) enlarged by the pixels are necessary for the luminance signal. Therefore, the horizontal and vertical sizes are 16 × 16, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, and 4 × 4 block sizes for motion compensation horizontally and vertically, respectively. 21 × 21, 21 × 13, 13 × 21, 13 × 13, 13 × 9, 9 × 13, and 9 × 9 pixel regions that are enlarged by 5 pixels are required.

  The color difference signal is generated by linear interpolation from four integer precision pixels around the decimal precision pixel. For example, in the case of 4: 2: 0 format, the motion compensation block size of the color difference signal is 8 × 8, 8 × 4, 4 × 8, 4 × 4, 4 × 2, 2 × 4, 2 × 2, and is a decimal number. The reference pixel areas necessary for obtaining the precision pixels are 9 × 9, 9 × 5, 5 × 9, 5 × 5, 5 × 3, 3 × 5, 3 ×, respectively, enlarged horizontally and vertically by one pixel. 3

  As described above, there arises a problem that the data transfer amount of the reference pixel data necessary for performing motion compensation increases. Therefore, in order to read and process the reference image data at a high speed, a high-performance and expensive memory is required, and the cost of the image decoding device increases.

  In order to solve the above problem, Patent Document 1 discloses a technique for providing a buffer for temporarily storing pixels acquired from a reference picture and performing motion compensation without re-acquiring pixels already stored in the buffer. Yes. This technique uses the fact that once acquired pixels are likely to be reused in motion compensation of other blocks, and reduces the amount of data acquired from the image memory.

JP 2005-354673 A

H. H.264 ISO / IEC 14496-10 standard

  However, the above conventional configuration is efficient because the data stored in the buffer can be used when the correlation between the motion vectors for each macroblock is large. However, if the motion vector is random, the buffer is buffered. You cannot use the data inside. Therefore, H.H. There is a problem that a wide-band memory is required to perform decoding processing on all encoded data compliant with the H.264 standard without failure.

  An object of the present invention is to provide an image decoding apparatus that can decode any encoded data without fail even when a narrow-band memory is used.

  A first invention of an image decoding device according to the present invention uses an image memory for storing decoded image data, a reference image acquisition unit for acquiring a reference picture necessary for motion compensation from the image memory, and the reference picture A motion compensation unit that performs motion compensation processing, and an image output unit that reads and outputs an output image from the image memory, and the image memory of image data obtained by decoding a decoding target picture within a preset target processing time A controller that controls whether or not the decoding process of the picture to be decoded is interrupted and the decoding process is resumed from the next picture.

  In the second invention, the control unit replaces the image data of the area in the screen that has not been decoded with the image data of the picture that was output immediately before the picture for which the decoding process has been interrupted. It is characterized by that.

  According to a third aspect of the present invention, the control unit performs the interruption of the decoding process only when a decoding target picture cannot be a reference picture of another picture.

  According to the first aspect of the present invention, it is determined whether or not the processing is completed within the target processing time in units of pictures, and if not completed, the encoding processing is terminated halfway. As a result, even if the memory bandwidth required for reading a reference image for motion compensation processing exceeds the memory bandwidth used for a certain picture in the moving image encoded data, the effect is limited to one picture. Thus, there is no delay in the decoding process of the entire encoded data, and the influence in the time axis direction can be minimized.

  According to the second invention, a region that has not been decoded due to the decoding process being aborted is replaced with the immediately preceding picture having a large correlation. As a result, it is possible to minimize the influence of image disturbance in the picture.

  According to the third invention, the picture for which the decoding process is terminated is limited to a picture that cannot be another reference picture. As a result, the influence caused by the decoding process abortion is not propagated to other pictures, but remains in the picture.

The block diagram which shows the structure of the image decoding apparatus in embodiment Timing chart explaining operation of image decoding apparatus in embodiment The block diagram which shows the structure of the conventional image decoding apparatus Illustration of motion compensation block Explanatory drawing of pixel interpolation processing by 6-tap filter Illustration of reference image data reading area

  Embodiments will be described below with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an image decoding apparatus according to an embodiment. FIG. 2 is a diagram for specifically explaining the operation. In FIG. 1, an image decoding apparatus 100 includes an input terminal 101, a variable length decoding unit 102, an inverse quantization unit 103, an inverse orthogonal transform unit 104, an adder 105, an intra prediction unit 106, a signal selection unit 107, and a motion compensation unit. 108, a reference image acquisition unit 109, a control unit 110, a memory controller 111, an image memory 112, an image output unit 113, and an output terminal 114.

  The variable length decoding unit 102 receives the H.264 input from the input terminal 101. The variable-length code of the H.264 image encoded data is decoded, and the variable-length decoding result is output in a macro block unit of 16 × 16 pixel size. The output includes quantized transform coefficient data, quantization parameter information, macroblock type information indicating a macroblock coding mode, motion information indicating a motion vector and a reference picture, and the like.

  The quantized transform coefficient data is subjected to an inverse quantization process by the inverse quantization unit 103 and an inverse orthogonal transform process by the inverse orthogonal transform unit 104. The inverse orthogonal transform unit 104 outputs prediction difference data.

  When the macroblock is subjected to intraframe prediction encoding, the intraframe prediction unit 106 generates intraframe prediction data from the surrounding pixel data of the macroblock according to the prediction mode indicated by the macroblock type information.

  When the macroblock is motion prediction encoded, the reference image acquisition unit 109 generates a reference picture rectangle necessary for generating motion prediction data from information indicating a reference picture and a motion vector for each motion compensation block. An area is obtained, a corresponding address on the image memory 112 is determined, and a read request is issued to the memory controller 111.

  The motion compensation unit 108 performs interpolation filter processing using the reference picture data acquired via the reference image acquisition unit 109, and generates motion prediction data with 1/4 pixel accuracy.

  The signal selection unit 107 selects intra prediction data or motion prediction data according to the prediction mode of the macroblock. The selected prediction data and the prediction difference data output from the inverse orthogonal transform unit 104 are added by the adder 105 to generate decoded data of the macroblock. The decoded data is written into the image memory 112 via the memory controller 111.

  The image output unit 113 requests the memory controller 111 to read a necessary picture in accordance with the reproduction image output timing, and outputs the read data.

  The operation of the memory controller 111 will be described. In this embodiment, the image memory 112 is configured by a DDR-SDRAM (Double-Data-Rate Synchronous Dynamic Random Access Memory). Note that the image memory 112 is not limited to the DDR-SDRAM, and may be a memory of another form as long as it can hold an image. A circuit (master) for accessing the memory is composed of a total of three circuits: (1) decoding data writing, (2) reproduction output reading, and (3) reference image reading. The memory controller 111 performs arbitration processing of memory access requests from each master, and performs write and read access of the selected master. The memory controller 111 performs mode setting and refresh operation for the DDR-SDRAM.

  Of the access requests to the memory controller 111, the amount of data per picture is constant for writing decoded data and reading image output. On the other hand, the amount of data for acquiring a reference image differs depending on the encoding mode. For example, when the entire picture is intra-coded like an I picture, the read data amount of the reference image is zero. On the other hand, when encoding is performed by unidirectional prediction with a motion compensation block size of 16 × 16, it is necessary to read out 21 × 21 = 441 pixels in the luminance signal. Further, when encoding is performed with bi-directional prediction with a motion compensation block size of 4 × 4, it is necessary to read out a reference image of 9 × 9 size per motion compensation block, and the number of motion compensation blocks per macroblock is 16, respectively. Since twice the number of blocks is required for the direction prediction, it is necessary to read out 9 × 9 × 16 × 2 = 2592 pixels per macroblock of 16 × 16 pixels. In such a macroblock, the time for reading data from the image memory 112 becomes long. That is, a picture with a large proportion of macroblocks with a small motion compensation block size requires a longer time for decoding processing than a picture with no macroblock.

  The control unit 110 monitors the progress of the decoding process in units of pictures from the acquisition state of the reference image data, and when the predetermined target time is exceeded, the control unit 110 cancels the decoding process of the picture with the variable length decoding unit 102 and the image. Notify the output unit 113. The variable length decoding unit 102 stops the decoding process after the macroblock at the time when the decoding process abort is notified in the picture being decoded, and restarts the decoding process from the first macroblock of the next picture.

  The image output unit 113 normally reads out and outputs reproduced image data from the image memory 112 for each picture. If the decoding process is interrupted, the picture data output immediately before is read from the image memory 112 and output for the undecoded area.

  FIG. 2 is an explanatory diagram of the decoding abort process. In FIG. 2, (a) shows the decoding processing timing for each picture. The rectangle in (a) is each decoded picture, and the characters in the rectangle are the picture type and display order. For example, I2 means the picture to be displayed second in the I picture. (B) is a write picture address on the image memory 112, (c) is reproduction data read by the image output unit 113, (d) is a read picture address, and (e) is a screen image of the reproduction image.

  The dotted line in FIG. 2A indicates the target decoding completion time for each picture. The control unit 110 forcibly terminates the decoding process when each picture does not complete decoding by the target decoding completion time. The decoding process is terminated for the B1 picture in the figure. When the image output unit 113 reads and outputs data from the image memory 112 for each picture, the image output unit 113 reads the data from the image memory 112 while changing the read address so that an undecoded area outputs the previous picture. In FIG. 2C, the hatched portion of the playback output is the section where the data of the previous picture is output, and the playback image replaces the hatched area at the bottom of the screen with the previous picture as shown in FIG. Displayed.

  As described above, according to the present embodiment, for each picture, it is determined whether or not the processing is completed within the target processing time. Even when there is a large amount of data required to read the reference image depending on the encoding conditions and a picture that requires a read bandwidth exceeding the memory performance is present in the encoded data, the decoding process of the entire encoded data is delayed. Does not occur, and its effect is limited only within the picture.

  Also, by replacing the area that has not been decoded with the picture that was output immediately before, even if the decoding process is interrupted halfway, the reproduced image of that picture can be reproduced without being greatly disturbed.

  Further, H.C. In the H.264 standard, a flag nal_ref_idc in a stream indicates whether the picture is a reference picture of another picture. By looking at this flag, it is possible to perform the censoring process only when it is not a reference picture. Thereby, it is possible to prevent the influence of decoding truncation from propagating to other pictures. In bi-directional prediction, the reference picture is decoded before the non-reference picture preceding in the display order, so that there is a large time margin from the completion of decoding to display, and playback is performed without performing the decoding truncation process. No bankruptcy will occur.

  In the present embodiment, the configuration in which the control unit 110 and the memory controller 111 are separately provided has been described. However, the memory controller 111 may not be provided, and the control unit 110 may control the image memory 112. .

  The image decoding apparatus according to the present embodiment can keep the performance required for the memory required for processing low, and is useful for high-resolution image signal encoding systems and low-cost image decoding apparatuses.

DESCRIPTION OF SYMBOLS 102 Variable length decoding part 103 Inverse quantization part 104 Inverse orthogonal transformation part 106 Intra-screen prediction part 108 Motion compensation part 109 Reference image acquisition part 110 Control part 111 Memory controller 112 Image memory 113 Image output part

Claims (4)

An image decoding device that decodes encoded data obtained by encoding a moving image by a decoding process with motion compensation,
An image memory for storing the decoded image data;
A reference picture acquisition unit for acquiring a reference picture necessary for motion compensation from the image memory;
A motion compensation unit that performs motion compensation using the reference picture;
An image output unit that reads and outputs an output image from the image memory;
It is determined whether or not the writing of the image data obtained by decoding the decoding target picture within the target processing time set in advance to the image memory is completed. If it is determined that the writing is not completed, the decoding process of the decoding target picture is interrupted. A control unit that controls to resume the decoding process from the next picture;
An image decoding apparatus comprising: The controller is
The image decoding apparatus according to claim 1, wherein the image data of the area in the screen that has not been decoded is replaced with the image data of the picture that was output immediately before the picture for which the decoding process has been interrupted. The controller is
The image decoding apparatus according to claim 1, wherein the decoding process is interrupted only when a decoding target picture cannot be a reference picture of another picture. An image decoding method for decoding encoded data obtained by encoding a moving image by a decoding process with motion compensation,
Storing the decoded image data in an image memory;
Obtaining a reference picture required for motion compensation from the image memory;
Performing motion compensation using the reference picture;
Reading and outputting an output image from the image memory; and
It is determined whether or not the writing of the image data obtained by decoding the decoding target picture within the target processing time set in advance to the image memory is completed. If it is determined that the writing is not completed, the decoding process of the decoding target picture is interrupted. Controlling to resume the decoding process from the next picture;
An image decoding method including:

Images