JP2004007736A - Device and method for decoding image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image decoding device having error resistance property and performing transmission with little delay, and to provide a method for decoding an image. <P>SOLUTION: An image decoding device is provided with an in-frame decoding means and an inter-frame decoding means. The in-frame decoding means applies in-frame decoding to encoded data which is identified as in-frame encoded data, based on identifying information, from a bit stream consisting of image data comprising a plurality of kinds of encoded data, and of the identifying information for obtaining correlation between an in-frame decoded image and an inter-frame decoded image, and also for indicating a reference image determined based on the correlation. The inter-frame decoding means applies inter-frame decoding to the encoded data which is identified as the inter-frame encoded data, based on the identifying information for deeming the encoded data decoded by the in-frame decoding means as reference data, in parallel with the operation of the in-frame decoding means. The device includes: a template storage memory 24; a frame delay memory 23; a Huffman decoder 116; an inverse quantizer 107; an inverse DCT 108; a multiplex movement vector decoder 29; and an adder 109, etc. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image decoding apparatus and method for encoding and decoding moving image data in, for example, a videophone.
[0002]
[Prior art]
In recent years, in order to transmit or record moving image data in videophones, video conference systems, and the like, image compression techniques have been required to achieve higher compression ratios with lower bit rates. I have. These technologies include MPEG1 / 2 in ISO / IEC and H.264 in ITU-T. 261, H .; 263 etc. are standardized.
[0003]
FIG. 17 is a block diagram of a conventional image encoding apparatus, and FIG. 18 is a block diagram of an image decoding apparatus corresponding to the encoding apparatus. 263 is realized.
[0004]
In FIG. 17, when an input image is intra-frame encoded, the intra-frame / inter-frame encoding changeover switch 11 is switched upward. The input image is subjected to a discrete cosine transform by the DCT 5, quantized by the quantizer 6, further variable-length coded by the Huffman coder 12, multiplexed by the MUX (multiplexer, multiplexing unit) 14, and Output as a stream. At this time, a part of the signal quantized by the quantizer 6 is inversely quantized by the inverse quantizer 7, further subjected to inverse discrete cosine transform by the inverse DCT 8, and transmitted to the frame delay memory 3 through the adder 9 as a reference image. Is stored. In this example, the Advanced Motion Compensation Mode, that is, the unit of the encoding process is an 8 × 8 block unit.
[0005]
On the other hand, when the input image is inter-frame encoded, the intra-frame / inter-frame encoding switch 11 is switched to the lower side. The input image is compared with a reference image stored in a frame delay memory 3 by a motion estimator 1 to detect a motion vector for each block, and stored in a motion vector memory 2. The motion compensator 4 generates a predicted image from the reference image by searching for an area corresponding to the block in the reference image for each block based on the motion vector. That is, motion compensation is performed on the reference image. The residual between the prediction image thus generated and the input image is obtained by the subtractor 10, the residual signal is encoded through the DCT 5 and the quantizer 6, and further variable-length encoded by the Huffman encoder 12. The data is multiplexed by the MUX 14 and output as a bit stream. At this time, the quantized signal is inversely quantized by the inverse quantizer 7, further subjected to inverse discrete cosine transform by the inverse DCT 8, further added to the predicted image output from the motion compensator 4 by the adder 9, and It is stored in the delay memory 3 as a reference image. The motion vector obtained by the motion estimator 1 is encoded by the motion vector encoder 13 and then multiplexed by the MUX 14 with the residual signal output from the Huffman encoder 12 and output.
[0006]
Next, in FIG. 18, when the bit stream encoded and output by the above-described image encoding device is input to the image decoding device, the DMUX (demultiplexer, separation unit) 15 outputs each of the image signal and the motion vector. It is separated into encoded signals. The image signal is decoded by a Huffman decoder 16 and further decoded by an inverse quantizer 7 and an inverse DCT 8. At this time, if the image signal is intra-frame encoding, the intra-frame / inter-frame encoding changeover switch 18 is connected to the upper side, and is output as it is as an output image. This output image is stored in the frame delay memory 3 as a reference image.
[0007]
On the other hand, the motion vector separated by the DMUX 15 is decoded by the motion vector decoder 17 and stored in the motion vector memory 2. Further, a predicted image is generated from the reference image of the frame delay memory 3 by the motion compensator 4 based on the motion vector, and the generated predicted image is added by the adder 9 to the image signal output from the inverse DCT 8. You. At this time, if the image signal is an inter-frame coded signal (that is, a residual signal), the intra-frame / inter-frame coding changeover switch 18 is connected to the lower side, and the added signal is output as an output image. Is done.
[0008]
Here, as shown in FIG. 19A, the motion estimator 1 searches the reference image for a region having the strongest correlation with the target block for each of the blocks obtained by dividing the input image into 8 × 8 blocks. , The motion vector is obtained by obtaining the deviation. At this time, for example, as shown in FIG. 19B, the area in the reference image corresponding to blocks T, B, L, and R around a certain block C partially overlaps with the area corresponding to block C, Or they may be separated at intervals. As a result, when a predicted image is formed from each of the searched areas, the image is degraded due to overlapping or discontinuous portions. Therefore, in order to prevent such deterioration in image quality, in order to take into account pixels around the target block, the prediction image is corrected using the motion vectors of the upper, lower, left, and right blocks of the target block. That is, as shown in FIG. 19B, in order to obtain the predicted value of the block C, the reference values corresponding to the five blocks are obtained from the motion vector of the block C and the upper, lower, left, and right blocks T, B, L, R. Each area in the image is obtained, pixels are read from the five areas, multiplied by a coefficient set for each block shown in FIG. 20, added, and finally divided by 8 for normalization. Similar processing is performed for each block, and a predicted image is obtained using the same. The neighborhood motion vector shown in the output of the motion vector memory 2 in FIGS. 17 and 18 indicates this.
[0009]
According to the above method, discontinuous portions occurring in an image are alleviated and the image quality is improved. Also, when the prediction error (residual) cannot be coded sufficiently at a low bit rate, only the motion vector is transmitted and only the predicted image is transmitted. Therefore, a sharp image is obtained in a portion where the motion is uniform, and a smooth image is obtained in a portion where the motion is uneven.
[0010]
[Problems to be solved by the invention]
However, in the conventional method as described above, since a predicted image is generated using the motion vectors of the upper, lower, left, and right blocks around the target block, matching cannot be achieved unless the same reference image is used, and the image to be referred to is the previous frame. However, there is a problem of lack of expandability.
[0011]
Further, for example, in MPEG1 / 2, the P image data can be encoded independently of the B image by the IBBBPBBPBBP structure. However, in MPEG1 / 2, the bit stream is not defined by being multiplexed, and is sequentially transmitted. Therefore, when a bit stream of a template (representative frame) is constituted by a CPU or a DSP when a bit of the template (representative frame) is huge, the bit stream is not defined within a predetermined time. In some cases, the frame cannot be decoded.
[0012]
FIG. 21 is a diagram showing the timing of the decoding process in the conventional image decoding device. 21, t0 and t1 indicate templates, and p0, p1,... P6 indicate general frames. In the upper diagram, since the decoding processing of the templates t0 and t1 does not take much time and is in time for the display timing, the frames are sequentially displayed from p0. On the other hand, in the lower diagram, the decoding process of the template t1 (p4) takes time and cannot be performed in time for the display timing of p4. At this time, since the decoding processes of p3 and p4 cannot be skipped as a matter of course, a margin is provided in the buffer until the frame display timing, and when the decoding process of p4 is delayed in advance, the decoding processes of p5 and p6 are skipped. This problem can be dealt with by performing the decoding processing of the next template.
[0013]
However, in the method of transmitting a single bit stream as described above, the bits must be interpreted one by one until the next template bit stream comes, and the processing time can be sufficiently reduced even if the processing of the general frame is skipped. There is a problem that can not be done.
[0014]
An object of the present invention is to provide an image decoding apparatus and an image decoding method capable of performing error-resistant and low-delay image transmission in consideration of such problems in conventional image decoding.
[0015]
[Means for Solving the Problems]
In order to achieve the above object, a first aspect of the present invention provides image data including a plurality of encoded data generated by encoding an image, the intra-frame decoded image and the inter-frame decoded image. From the bit stream including the identification information indicating the reference image determined based on the correlation, the encoded data identified as the intra-frame encoded data based on the identification information, Intra-frame decoding means to be decoded, and in parallel with the operation of the intra-frame decoding means, the encoded data decoded by the intra-frame decoding means as reference data, and An inter-frame decoding means for inter-frame decoding encoded data identified as inter-coded data, wherein the intra-frame decoding is performed. By the time the stage ends the intra-frame decoding of the encoded data, the inter-frame decoding means refers to the intra-frame encoded data whose display order is earlier than the predetermined encoded data. When the inter-frame decoding of the intra-frame encoded data encoded as inter-frame data is not completed,
The image decoding method, wherein the operation of the inter-frame decoding by the inter-frame decoding unit is performed with priority over the operation of the intra-frame decoding unit decoding the intra-frame encoded data. Device.
[0016]
In addition, the second present invention relates to a method of calculating a correlation between image data including a plurality of encoded data generated by encoding an image, the image, an intra-frame decoded image, and an inter-frame decoded image. And a frame for decoding intra-frame encoded data identified as intra-frame encoded data based on the identification information from a bit stream including identification information indicating a reference image determined based on the correlation. Inner decoding step;
Performed in parallel with the intra-frame decoding step, the encoded data decoded in the intra-frame decoding step as reference data, identified as inter-frame encoded data based on the identification information An inter-frame decoding step of inter-frame decoding of the encoded data,
Until the intra-frame decoding step of the encoded data is completed, in the inter-frame decoding step, an intra-frame encoded data whose display order is earlier than the encoded data in the inter-frame decoding step is used as the reference data. An image decoding method characterized by performing the inter-frame decoding step prior to the intra-frame decoding step if the decoding step of the encoded intra-frame encoded data is not completed. .
[0017]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, the present invention will be described with reference to the drawings showing the embodiments.
(Embodiment 1)
FIG. 1 is a block diagram illustrating a configuration of an image encoding device according to a first embodiment of the present invention, and FIG. 2 illustrates a configuration of an image decoding device corresponding to the image encoding device. It is a block diagram. That is, the image encoding apparatus includes a frame delay memory 23 for storing a reference image, a motion estimator 21a as a motion detection unit for detecting a motion vector based on the reference image and the input image, A weighted motion compensator 22a for generating a predicted image from a reference image is provided, a template storage memory 24 for storing a template (representative frame) as a reference image, and a template (a reference different from the reference image of the frame delay memory 23). And a weighted motion compensator 22b that generates a predicted image from a reference image based on the motion vector.
[0018]
The correlation comparator 26 compares the degree of correlation between the reference image and the input image using the motion vectors detected by the motion estimators 21a and 21b. As a result of the comparison, the reference image having the higher correlation is obtained. , A reference image changeover switch 27 as a changeover means for selecting a weighted motion compensator, a predicted image memory 25 for storing a predicted image output via the reference image changeover switch 27, a predicted image of the predicted image memory 25, A subtractor 10 for obtaining a residual from an input image is provided.
[0019]
Also, an intra-frame / inter-frame encoding changeover switch 11 that switches between input images depending on whether it is intra-frame encoding or inter-frame encoding, and a DCT 5 that performs discrete cosine transform of an image signal from the intra-frame / inter-frame encoding switching switch 11. , A quantizer 6 for quantizing the discrete cosine-transformed signal, a Huffman encoder 12 for variable-length encoding the quantized signal, and an output of a correlation comparator 26, ie, switching information. Multiplexed motion vector encoder 28 that multiplexes and encodes a switching control bit as, and a motion vector selected by the switching control bit among the motion vectors from the motion estimators 21a and 21b. The output of the vector encoder 28 and the output of the Huffman encoder 12 are multiplexed to form a bit stream. MUX14 are provided for outputting a beam.
[0020]
Further, an inverse quantizer 7 for inversely quantizing the output of the quantizer 6, an inverse DCT 8 for performing an inverse discrete cosine transform of the inversely quantized signal, and storing the inverse discrete cosine transformed signal and the prediction image memory 25. An adder 9 for adding the predicted image thus obtained to generate a reference image for the next input image and a template update switch 19 for updating the reference image in the template storage memory 24 are provided.
[0021]
Here, the subtractor 10, the DCT 5, the quantizer 6, and the like constitute an encoding unit, and the inverse quantizer 7, the inverse DCT 8, the adder 9, and the like constitute a decoding unit.
[0022]
On the other hand, the image decoding apparatus shown in FIG. 2 receives the bit stream output from the image encoding apparatus shown in FIG. 1 and separates it into an image signal (usually a residual signal) and a multiplex motion vector. A Huffman decoder 16 for decoding an image signal, an inverse quantizer 7 for inversely quantizing the decoded signal, and an inverse DCT 8 for inverse discrete cosine transform of the inversely quantized signal are provided. A multiplexed motion vector decoder 29 that decodes the separated multiplexed motion vector and separates the multiplexed motion vector into a motion vector and a switching control bit, and a weighted motion compensator that generates a predicted image from a reference image based on the separated motion vector. 22c, a predicted image memory 25 for storing the generated predicted image is provided.
[0023]
An adder 9 for adding the predicted image of the predicted image memory 25 and the output of the inverse DCT 8 to generate an output image; an intra-frame / inter-frame code that switches depending on whether the output image is intra-frame encoding or inter-frame encoding. There are provided a changeover switch 18, a frame delay memory 23 for storing an output image as a reference image, and a template storage memory 24 for storing a template among the output images as a reference image, and further for updating the reference image in the template storage memory 24. And a reference image changeover switch 27 for switching the reference image between the frame delay memory 23 and the template storage memory 24 according to a control bit.
[0024]
Here, the Huffman decoder 16, the inverse quantizer 7, the inverse DCT 8, the multiplex motion vector decoder 30 and the like constitute a decoding unit, and the adder 9 and the like constitute an image generation unit. Here, the detection of the motion vector is performed in units of 16 × 16 blocks, that is, in a so-called macroblock unit. The blocks referred to in the claims are blocks of a predetermined size, and include not only 8 × 8 blocks, but also macroblocks, and may be blocks of other sizes.
[0025]
Next, the operation of the image encoding and decoding apparatus according to the first embodiment will be described with reference to the drawings.
[0026]
First, the correspondence between a general frame and a representative frame (template) will be described. FIG. 3 shows the correspondence between general frames and templates. In FIG. 3, p0 to p6 indicate general frames, and t0 and t1 indicate templates. The template t0 is created by copying from the general frame p0, and the general frames p2 and p3 are created with reference to the template t0. The template t1 is created by copying from the general frame p4, and the general frame p6 is created with reference to the template t1. The general frame is stored in the frame delay memory 23, and the template is stored in the template storage memory 24 when updating the template.
[0027]
It is assumed that the previous frame is already stored in the frame delay memory 23 as a reference image, and the template is stored in the template storage memory 24 as a reference image. Here, when an input image arrives, the motion estimator 21a detects a motion vector in units of 16 × 16 macroblocks from the input image and the reference image of the frame delay memory 23, and outputs the detection result as a weighted motion compensator 22a. , The correlation comparator 26 and the multiplex motion vector encoder 28. Similarly, the motion estimator 21b detects a motion vector in macroblock units from the input image and the reference image in the template storage memory 24, and outputs the detection result as a weighted motion compensator 22b, a correlation comparator 26, and a multiplex motion vector code. Output to the converter 28.
[0028]
Next, based on the motion vector detected above, the weighted motion compensator 22a searches the reference image for a region having the highest correlation for each of the target macroblocks, and searches around the searched region. A region of 24 × 24 pixels including the macroblock is extracted, and the extracted pixels of the wide region are multiplied by a weighting factor as shown in FIG. 5 to obtain the pixels of the extracted region. Similarly, the weighted motion compensator 22b obtains pixels in an area of 24 × 24 pixels using the template as a reference image. Here, the weight coefficients on the left side of FIG. 5 are for the luminance signal, and the weight coefficients on the right side are for the color difference signal.
[0029]
On the other hand, the correlation comparator 26 compares each output of the motion estimators 21a and 21b to determine which reference image and the input image have a higher correlation, and sets the reference image switch 27 according to the result. By switching, the output of the weighted motion compensator to be connected to the predicted image memory 25 is selected. At this time, the comparison result is also output to the multiplex motion vector encoder 28 as a switching control bit. As a result, assuming that the weighted motion compensator 22a is selected, as shown in FIG. 4, in the predicted image memory 25, from the previous frame stored in the frame delay memory 23, a weighted wide area (indicated by a circle, (Dotted squares indicate macroblocks). Thereafter, when the correlation between the input image and the template is determined to be high by the correlation comparator 26 and the weighted motion compensator 22b is selected, as shown in FIG. 4, the pixels in the weighted area are input from the template. You. In this way, the processing is performed on all the macroblocks, and the superimposed pixels are further divided by 8 for normalization, whereby a predicted image is generated in the predicted image memory 25. This normalization is specifically performed by shifting the data from the predicted image memory 25 by 3 bits and reading out the data, so that special hardware is not required. As described above, since it is not necessary to use a motion vector around the target block for one predicted image, it is possible to use a different reference image for each block. The weight of the pixel of the overlapping portion (4 pixels in this example) of the divided area can be weighted.
[0030]
In this case, the block boundaries are smoothly connected by generating a predicted image from a region of a previous frame that is close in time with a large motion vector and a still image portion with a small motion vector from a high-resolution template. Can be generated.
[0031]
In addition, by generating a prediction image using a reference image as a background image and a previous frame, it is possible to smoothly predict a background portion which is hidden between the background portion and the preceding moving object.
[0032]
Also, when the reference image is a preceding and succeeding frame, it can be applied to a B frame structure performed by MPEG1 / 2.
(Embodiment 2)
FIGS. 6 and 7 are block diagrams showing the configuration of the image encoding device according to the second embodiment of the present invention. FIG. 8 is a block diagram showing a configuration of an image decoding device corresponding to the image encoding device. This embodiment is significantly different from the above-described first embodiment in that a general frame and a template are separately processed, and data is packetized into separate bit streams. Is that an error detection code is added and an error correction code is added to the template.
[0033]
In FIG. 6, the image coding apparatus includes an input switch 30 for switching an input image between a general frame and a template, an output buffer 31 for a general frame, and data for a general frame. A packetizer 32 with an error detection code for adding an error detection code and performing packetization, and a MUX 33 for multiplexing and outputting packet data of a general frame and packet data of a template are added. Note that the template update switch 19 in FIG. 1 has been removed.
[0034]
Also, in FIG. 7, when the input image is a template, it is a portion for encoding the image, an intra-frame / inter-frame encoding changeover switch 41, a DCT 43 for performing a discrete cosine transform of an image signal, and a signal subjected to the discrete cosine transform. 44, a Huffman encoder 48 for variable-length encoding the quantized signal, a template storage memory 24 for storing a template as a reference image, and an input image and a template storage memory. A motion estimator 21c for detecting a motion vector based on the template stored in 24, a motion compensator 4 for generating a predicted image from the template based on the detected motion vector, and taking a residual between the predicted image and the input image Subtractor 42, inverse quantizer 45 for inversely quantizing the output of quantizer 44, and its inverse quantization Conversely DCT46, adder 47 for adding the predictive image and its inverse cosine transformed signal is provided to inverse discrete cosine transform signal. Further, a motion vector encoder 49 for encoding the motion vector detected by the motion estimator 21c, a MUX 50 for multiplexing the encoded motion vector and the variable length encoded signal, an output buffer 51 for template, An error correction code-added packetizer 52 for adding an error correction code to the template data and performing packetization is provided.
[0035]
On the other hand, in FIG. 8, the image decoding apparatus is provided with a DMUX 60 for receiving the bit stream from the image encoding apparatus and separating the packet into general frame packets and template packets. , A packet decomposer 62 with error detection for decomposing packets of a general frame, an input buffer 61 for a general frame, and an image decoding device substantially similar to FIG. 2 are provided.
[0036]
For processing of the template packet, an error correcting packet decomposer 63 for decomposing the template packet, an input buffer 64 for the template, a DMUX 115 for separating the signal from the input buffer 64 into an image signal and a motion vector, Huffman decoder 116 and inverse quantizer 107 and inverse DCT 108 for decoding the separated image signal, template storage memory 24 for storing a template as a reference image, motion vector decoder 17 for decoding a motion vector, and decoding thereof A motion vector memory 2 for storing the generated motion vector, a motion compensator 4 for generating a predicted image from a template in a template storage memory 24 based on the motion vector, and adding the generated predicted image to an output of the inverse DCT 108 Adder 109 in the frame Interframe coding changeover switch 118 is provided. Here, the decoding process of the template is performed by a method similar to the conventional method.
[0037]
Further, a switching control bit output from the multiplex motion vector decoder 29 causes a reference image switch 65 for switching between a general frame and a template in the frame delay memory 23 as a reference image, and a packet decomposer 62 with error detection to convert the general frame into a general frame. An output image switch 66 is provided for switching the output image from a general frame to a template when an error is detected.
[0038]
Next, the operation of the image encoding and decoding apparatus according to the second embodiment will be described with reference to the drawings.
[0039]
First, in the present embodiment, as shown in FIG. 9, the general frames p0 to p6 and the templates t0 and t1 identical to p0 and p4 are processed as separate bit streams. Also, the data is packetized, and the structure of the packet is, as shown in FIG. 11, provided at the beginning with an identifier for identifying a template or a general frame, and then provided with a frame number indicating a reproduction timing. Next, in the template, data and a Reed-Solomon error correction code are provided, and in a general frame, a frame number and a CRC error detection code of the template referred to by the general frame are provided. Here, the reason why the error correction code is attached only to the template and the error detection code is attached to the general frame is that the data amount of the error correction code is larger than the data amount of the error detection code.
[0040]
In FIG. 6, when the input image is a general frame, the input changeover switch 30 is connected to the upper side, and the same encoding processing as in the first embodiment is performed on the general frame. At this time, the quantizer 6 is controlled by signals from the output buffers 31 and 51. Here, the output buffer is a FIFO buffer, and is realized as a ring buffer that circulates a write destination address and a read destination address in a memory area. The buffer margin is expressed as the difference between the write destination address and the read destination address. When the write destination address exceeds the read destination address, the buffer overflows (the address moves cyclically, so it is not a simple size of the address). In this case, part of the transmission image information is lost. In order to avoid this, the quantization step may be changed according to the buffer margin, and accordingly, the quantizers 6 and 44 may be controlled based on the buffer margins of the output buffers 31 and 51. This process is called rate control. The image data variable-length coded by the Huffman coder 12 and the motion vector and the switching control bits multiplexed by the multiplex motion vector coder 28 are multiplexed by the MUX 14 and then output to the output buffer 31. The packetizer 32 with the error detection code creates the packet to which the CRC error detection code as shown in FIG. 11 is added.
[0041]
On the other hand, when the input image is a template, the input changeover switch 30 is connected to the lower side, and the encoding process is performed by the encoding unit for template shown in FIG. Here, the encoding unit performs the same encoding processing as that of a conventional general image encoding device. That is, when the template is intra-frame encoding, the intra-frame / inter-frame encoding changeover switch 41 is connected to the upper side, and is encoded by the DCT 43, the quantizer 44, and the Huffman encoder 48. When the template is inter-frame encoding, the intra-frame / inter-frame encoding switch 41 is connected to the lower side, and the motion estimator 21c detects a motion vector from the input template and the reference image in the template storage memory 24. Then, based on the detected motion vector, the motion compensator 4 generates a predicted image from the reference image, takes the residual between the predicted image and the input template by the subtractor 42, and outputs the residual signal. Encoding is performed by the DCT 43, the quantizer 44, and the Huffman encoder 48. The detected motion vector is encoded by the motion vector encoder 49.
[0042]
Next, the image data of the template variable-length coded by the Huffman coder 48 and the motion vector coded by the motion vector coder 49 are multiplexed by the MUX 50 and output to the output buffer 51, The packetizer 52 with the error correction code creates the packet to which the Reed-Solomon error correction code is added as shown in FIG.
[0043]
The packet with the error detection code of the general frame and the packet with the error correction code of the template created as separate bit streams as described above are output as a time-division multiplexed bit stream by the MUX 33.
[0044]
Next, in FIG. 8, the bit stream from the image encoding device is input to the DMUX 60, and is separated into general frame packets and template packets. This separation can be easily performed by the identifier added to the head of the packet. The packet of the separated general frame is decomposed by the packet decomposer with error detection 62 and the error detection code is interpreted. After the packet is decomposed, the data is output to the input buffer 61 and subjected to the same decoding processing as in the first embodiment. The packet of the template is decomposed by the packet decomposer 63 with error correction, and the error correction code is interpreted. After the packet is decomposed, it is output to the input buffer 64. Thereafter, the image signal and the motion vector are separated by the DMUX 115, and the image signal is decoded by the Huffman decoder 116, the inverse quantizer 107, the inverse DCT 108, and the adder 109, and the motion vector is stored in the template storage memory 24. The predicted image is input to the motion compensator 4 via the motion vector memory 2 in order to generate a predicted image from the reference image.
[0045]
Here, the decoding is performed with priority given to the processing of the template. For example, as shown in FIG. 10, when the template is encoded every four frames, the decoding process of the template t1 is performed with priority over the general frames p2 and p3 referring to the template t0 (see the upper diagram). At this time, as shown in the lower diagram, if the processing of the template t1 takes time and the display of p3 cannot be made in time even after decoding the general frame p3, the decoding processing of p3 is canceled in the middle. As a result, the decoding process at t1 can be completed by the display timing of the general frame p4.
[0046]
Further, in the present embodiment, when an error such as a transmission error is detected in a general frame by the packet decomposer 62 with error detection, the data of the general frame is discarded. That is, a control signal for the output image changeover switch 66 is output from the packet decomposer with error detection 62 in FIG. 8 and the output image is switched to the template. On the other hand, in the template, even if a transmission error or the like occurs, the data is error-corrected by the error correction code added to the packet. When an error occurs and the general frame is discarded, the template is forcibly output as a substitute frame for the general frame. As a result, even when there is an error in the transmission path or the like, it is possible to maintain the minimum image quality based on the template, and to make the transmission path or the like resistant to errors.
[0047]
In the above embodiment, the error correction code is a Reed-Solomon error correction code and the error detection code is a CRC error detection code. However, the present invention is not limited to this, and other error correction codes and error detection codes may be used. Of course, it is good.
(Embodiment 3)
FIG. 12 is a schematic diagram showing a correspondence between a processed image and a template according to the third embodiment of the present invention. This embodiment is an extension of the second embodiment described above. As shown in FIG. 12, a template in which general frames p0 to p6 refer to one template t0 has a larger panorama than a general frame. It is intended to be applied to a case of an image or the like or a case where a reference block can be specified in a reference image in advance. FIG. 13 is a diagram showing processing timing in the image shown in FIG. 12, and shows a state in which the decoding process of the general frame is sequentially performed from p0 during the decoding process of the template t0, and the frames are displayed.
[0048]
To realize this, for example, an upper left macroblock position and a lower right macroblock position (see FIG. 14) including a partial area of the template to be referred to are added to the head of the packet of the general frame. The upper left macro block position and the lower right macro block position occupied by the partial image to be transmitted with respect to the entire image are also added to the template packet so that the partial image necessary for decoding the template is decoded in advance. It is left to the encoding side to decompose the template image into what partial images. Encoding and decoding of the template are performed independently for the decomposed partial image, and assuming that the vertical and horizontal numbers of the macroblocks of the entire screen are known, the template decoding is performed based on the frame number and the upper left and lower right information of the macroblock. Can be uniquely reconstructed. Whether the reference area in the template is created at the correct timing also depends on the bit stream creation on the encoding side.However, on the decoding side, whether the playback of a general frame is possible depends on the packet of the general frame. Based on the frame number of a certain reference destination template and the reference macroblock range, the determination can be made from the head data of the packet.
[0049]
FIG. 15 is a block diagram illustrating a configuration of an image encoding device according to a fourth embodiment of the present invention. FIG. 16 is a block diagram showing a configuration of an image decoding device corresponding to the image encoding device. The image coding apparatus according to the present embodiment is different from the conventional image coding apparatus in that an offset map in which an offset value for changing a quantization step used for quantization for each block is arranged so as to correspond to an image. 73 and a DPCM Huffman encoder 74 for variable-length encoding the data of the offset map 73 are added. Here, the operation of the DPCM Huffman encoder 74 is based on H.264. The variable length code of each direction component of the motion vector 261 can be used. The quantizer 70 and the inverse quantizer 71 perform quantization and inverse quantization according to the offset map 73, and the MUX 72 includes the Huffman encoder 12, the DPCM Huffman encoder 74, and the motion vector encoder. Each output signal of the unit 13 is multiplexed to generate a bit stream to be transmitted.
[0050]
On the other hand, in the image decoding apparatus according to the present embodiment, the DMUX 80 separates the bit stream input from the encoding apparatus into image data, motion vectors, and offset map data. A DPCM Huffman decoder 81 for decoding the data of the offset map separated by the DMUX 80 and an offset map 82 for storing the output offset map value are added to the image decoding apparatus. Here, the inverse quantizer 83 changes the quantization step according to the offset map 82 and performs inverse quantization.
[0051]
Here, for example, an image of a videophone or the like, where there is almost no movement in the peripheral area and an insignificant background area sends a rough image, and an area that requires fine movement such as a change in facial expression of a person's face in the center is a fine image. , It is necessary to make the quantization at the center of the image finer. This is performed by adding an offset value of −2 to −3 in the central portion and +2 to +3 in the peripheral portion with respect to a general quantization step. A quantization step is sent for each macroblock, and offset map data is sent for each image. This offset is As a modification of H.263, one bit of control information named, for example, LQOM (load quantization step offset map) is added after PQUANT of the picture header, and the control information is used to reset, for example, an I picture, and P, B In other cases such as a picture, if the value is 0, the previous offset map is taken over as it is. Here, the default of the offset is all 0 (no offset).
[0052]
In the above embodiments, the predetermined size of each block is 16 × 16, and the predetermined size larger than the block size is 24 × 24. However, the present invention is not limited to this. In addition, other sizes may be used, and the wide predetermined size may be any size as long as it is larger than the size of the block.
[0053]
In the above-described embodiment, the means configured by dedicated hardware, such as a motion estimator, may be replaced with a similar function by software using a computer, if possible. .
[0054]
Note that another invention by the present inventor in the above embodiment is based on a motion detection unit that detects a motion vector for each block of a predetermined size from a reference image and an input image, and a motion detection unit that detects a motion vector based on the detected motion vector. An area having a predetermined size larger than a predetermined size including an area corresponding to each block of the input image is extracted from the reference image, and a predetermined weight is applied to each pixel of the wide area, and the weighting of the wide area is performed. A weighted motion compensator for generating a predicted image of an input image using each pixel, a predicted image memory for storing the predicted image, and a residual between the stored predicted image and the input image. The image encoding apparatus includes an encoding unit that encodes the difference, and a decoding unit that decodes the encoded image data to obtain a reference image.
[0055]
Also, the invention provides a decoding unit for decoding encoded data input from an image encoding device, a frame memory for storing a reference image, and each block of a predicted image based on a motion vector decoded by the decoding unit. An area of a predetermined size larger than the predetermined size including the area corresponding to the area is extracted from the reference image stored in the frame memory, and each pixel of the wide area is weighted in advance, and the weight of the wide area is calculated. A weighted motion compensating unit for generating a predicted image using each of the pixels, and an image generating unit for generating an output image based on the predicted image and a residual signal decoded by the decoding unit. Includes a decoding device.
[0056]
As described above, the reference image is not limited to the previous frame. For example, it is possible to use a different reference image to generate a predicted image.
[0057]
Also, the invention divides image data into a representative frame as a representative and a general frame other than the representative frame, and multiplexes the representative frame and the general frame as separate bit streams by adding frame identification information for identifying them. Includes an image encoding method for converting and transmitting.
[0058]
Further, the invention detects frame identification information from encoded data input from an image encoding device, and converts the input encoded data into representative frame and general frame image data according to the detected frame identification information. If the decoding of a general frame that refers to a representative frame that has already been decoded overlaps with the decoding of the representative frame, the decoding of the general frame is canceled. Including methods.
[0059]
As described above, even if it takes time to decode the representative frame, it is possible to easily skip the decoding process of the general frame.
[0060]
According to the invention, an error correction code is extracted from a representative frame of coded data input from an image coding apparatus, and when an error occurs in the data, correction coding is performed on the representative frame. An image decoding method for extracting an error detection code from a general frame, and discarding the general frame when an error occurs in the data, and replacing the discarded frame with a representative frame.
[0061]
As described above, for example, for a transmission path error, the minimum image quality can be maintained using the representative frame.
[0062]
According to the present invention, in order to change a quantization step for quantization of an image to be transmitted for each block, an offset map in which an offset value of the quantization step is set for each image, and data of the offset map are stored in an image. An image encoding device including an offset adding means for adding the data to the beginning of data.
[0063]
Further, the invention provides an offset extracting means for extracting an offset map from encoded data inputted from an image encoding device, and a quantization step for each image is changed based on the extracted offset value. Decoding means for dequantizing and decoding the encoded data input in the encoding step.
[0064]
As described above, the quantization step in each block can be easily changed according to the offset value of the offset map.
[0065]
The other inventions as described above have an advantage that, for example, an image to be referred to is not limited and an effect of scalable overlap motion compensation can be obtained.
[0066]
Also, in decoding an image for which the CPU and DSP have an indefinite (variable) processing time, there is an advantage that failure in image reproduction due to inadequate bitstream interpretation can be avoided.
[0067]
In addition, there is an advantage that image transmission with error tolerance and low delay can be performed by using a representative frame as a separate bit stream.
[0068]
【The invention's effect】
As is clear from the above description, the present invention has an advantage that image transmission with low error tolerance and low delay can be performed.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an image encoding device according to a first embodiment of the present invention.
FIG. 2 is a block diagram illustrating a configuration of an image decoding device corresponding to the image encoding device of FIG. 1;
FIG. 3 is a schematic diagram showing a correspondence between a processed image and a template according to the first embodiment.
FIG. 4 is a schematic diagram illustrating a relationship between a predicted image and a reference image according to the first embodiment.
FIG. 5 is a diagram illustrating an example of a weight coefficient according to the first embodiment.
FIG. 6 is a block diagram illustrating a part of a configuration of an image encoding device according to a second embodiment of the present invention.
FIG. 7 is a block diagram showing the remaining part of the configuration of the image encoding device leading to FIG. 6;
FIG. 8 is a block diagram illustrating a configuration of an image decoding device corresponding to the image encoding device according to the second embodiment.
FIG. 9 is a schematic diagram showing a correspondence between a processed image and a template according to the second embodiment.
FIG. 10 is a diagram for explaining a decoding processing timing of a multiplexed bit stream in the image decoding device according to the second embodiment.
FIG. 11 is a diagram showing a configuration of a packet according to the second embodiment.
FIG. 12 is a schematic diagram showing a correspondence between a processed image and a template according to a third embodiment of the present invention.
FIG. 13 is a diagram illustrating decoding processing timing in the image decoding device according to the third embodiment.
FIG. 14 is a diagram illustrating a reference area according to the third embodiment.
FIG. 15 is a block diagram illustrating a configuration of an image encoding device according to a fourth embodiment of the present invention.
FIG. 16 is a block diagram illustrating a configuration of an image decoding device corresponding to the image encoding device of FIG. 15;
FIG. 17 is a block diagram illustrating a configuration of a conventional image encoding device.
FIG. 18 is a block diagram illustrating a configuration of an image decoding device corresponding to the image encoding device of FIG. 17;
FIG. 19A is a diagram for explaining a method of obtaining a motion vector in a conventional image encoding device, and FIG. 19B is a diagram for explaining a method of obtaining a predicted image.
FIG. 20 is a diagram illustrating a method for obtaining a predicted image in a conventional image encoding device.
FIG. 21 is a diagram illustrating a decoding processing timing of a single bit stream in a conventional image decoding device.
[Explanation of symbols]
1 motion estimator
3,23 frame delay memory
4 Motion compensator
5 DCT
6 Quantizer
7 Inverse quantizer
8 Inverse DCT
21a, 21b motion estimator
22a, 22b, 22c Weighted motion compensator
24 Template storage memory
26 Correlation comparator
32 Packetizer with error detection code
52 Packetizer with error correction code
62 Packet Decomposer with Error Detection
63 Packet Decomposer with Error Correction
73, 82 Offset map
74 DPCM Huffman encoder
81 DPCM Huffman Decoder

Claims (2)

Image data including a plurality of encoded data generated by encoding an image, and a correlation between the intra-frame decoded image and the inter-frame decoded image, and a reference image determined based on the correlation. An intra-frame decoding unit for intra-frame decoding the encoded data identified as the intra-frame encoded data based on the identification information from a bit stream including the identification information indicating: In parallel with the operation of the above, the encoded data decoded by the intra-frame decoding means as reference data, the encoded data identified as inter-frame encoded data based on the identification information, the frame An image decoding apparatus comprising: an inter-frame decoding unit that performs inter-frame decoding.
By the time that the intra-frame decoding means completes the intra-frame decoding of the encoded data, the inter-frame decoding means performs intra-frame encoding whose display order is earlier than the predetermined encoded data. If the inter-frame decoding of the intra-frame encoded data obtained by inter-frame coding using the processed data as the reference data is not completed,
The image decoding method, wherein the operation of the inter-frame decoding by the inter-frame decoding unit is performed with priority over the operation of the intra-frame decoding unit decoding the intra-frame encoded data. apparatus. Image data including a plurality of encoded data generated by encoding the image, the image, the correlation between the intra-frame decoded image and the inter-frame decoded image, and determined based on the correlation An intra-frame decoding step of intra-frame decoding encoded data identified as intra-frame encoded data based on the identification information from a bit stream including the identification information indicating the reference image. Encoded data identified as inter-frame encoded data based on the identification information, with the encoded data decoded in the intra-frame decoding step performed in parallel with the decoding step as reference data. An inter-frame decoding step of inter-frame decoding data, comprising:
Until the intra-frame decoding step of the encoded data is completed, in the inter-frame decoding step, an intra-frame encoded data whose display order is earlier than the encoded data in the inter-frame decoding step is used as the reference data. An image decoding method, wherein when the decoding step of the encoded intra-frame coded data is not completed, the inter-frame decoding step is performed with priority over the intra-frame decoding step.

Images