JP2003333600A - Image encoding method and image decoding method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an inter-frame encoding method for selecting two optional reference frames from encoded frames to predict by interpolation, and efficiently encoding a block header denoting a block prediction type. <P>SOLUTION: A block header is variable-length-coded by using different code word tables between the case when a coding target frame includes a block subjected to interpolation prediction on the basis of a reference frame having a display time before and after display time information of the coding target frame in a picture and the case when the coding target frame includes a block subjected to interpolation prediction on the basis of a reference frame having a display time before and after the display time information of the coding target frame in a picture. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of encoding and decoding an image signal, and a recording medium having a program for implementing the same implemented by software.

[0002]

2. Description of the Related Art In recent years, with the development of multimedia applications, it has become common to uniformly handle information of all media such as images, voices and texts. At this time, it becomes possible to handle media in a unified manner by digitizing all media. However, since a digitized image has a huge amount of data, an image information compression technique is indispensable for storage and transmission. On the other hand, standardization of compression technology is also important in order to interoperate compressed image data. Image compression technology standards include ITU (International Telecommunication Union Telecommunication Standardization Sector) H.261, H.263, and ISO (International Organization for Standardization) MPEG (Moving Picture Experts Gro).
up) -1, MPEG-2, MPEG-4 and so on. Further, H.26L is currently being standardized by ITU as the latest image coding standard.

Interframe prediction with motion compensation is a technique common to these moving picture coding systems. In motion compensation of these moving image coding methods, a frame of an input image is divided into rectangles of a predetermined size (hereinafter referred to as blocks), and a prediction pixel is generated from a motion vector indicating a motion between frames for each block. To do. For inter-frame prediction of MPEG, frame 1 whose display time is earlier than the encoding target frame
Forward prediction that predicts from the number of frames, backward prediction that predicts from one frame whose display time is later than the frame to be encoded, frame that has a display time earlier than the frame to be encoded and a frame that has a display time later than the frame to be encoded. Bi-directional prediction is used that performs prediction by pixel interpolation from a single frame.

FIG. 11 is a block diagram showing the structure of a conventional image coding apparatus. The image coding device uses the image signal Im
Divide g into blocks and process each block.
The subtractor Sub subtracts the predicted image signal Pred from the image signal Img input to the image encoding device, and outputs it as a residual signal Res. The pixel decoding means ImgDec inputs the residual signal Res, performs image coding processing such as DCT conversion / quantization,
The residual coded signal ERes including the quantized DCT coefficient is output. The image decoding means ImgDec inputs the residual encoded signal ERes, performs image decoding processing such as inverse quantization / inverse DCT conversion, and outputs the residual decoded signal DRes. The adder Add adds the residual decoded signal DRes and the predicted image signal Pred, and outputs the reconstructed image signal Recon. With the reconstructed image signal Recon,
Signals that may be referred to in the subsequent inter-frame prediction are stored in the multi-frame buffer MFrmBuf.

Now, the MH picture will be described with reference to FIG. FIG. 12 is a conceptual diagram of an MH picture. Frm
Is the MH picture to be encoded, FwRef1 and FwRef2 are the encoded frames whose display time is earlier than the encoding target frame Frm, and B
wRef1 and BwRef2 indicate encoded frames whose display time is later than the encoding target frame Frm. Block Blk1 is a reference block RefBlk11 and frame Bw included in frame FwRef2.
It is a block that is inter-frame predicted with a prediction pixel generated by pixel interpolation from a reference block RefBlk12 included in Ref1. The block Blk2 is a block inter-frame predicted from the reference block RefBlk2 included in the frame FwRef1. The block Blk3 is a block inter-frame predicted from the reference block RefBlk3 included in the frame BwRef1. A block, such as the block Blk1, which is inter-frame predicted from a plurality of reference frames by pixel interpolation is called an interpolation prediction block. A block, such as the block Blk2, which is inter-frame predicted from a reference frame whose display time is previous to the encoding target frame is called a forward prediction block. A block, such as the block Blk3, which is inter-frame predicted from a reference frame whose display time is later than the encoding target frame is called a backward prediction block. Although not shown, a block in which inter-frame prediction is not performed and prediction is performed from pixels in the encoding target frame is referred to as an in-plane prediction block.
An MH picture is a picture that can include a block that performs inter-frame prediction from a plurality of reference frames, such as an interpolation prediction block, in the picture. On the other hand, a picture that does not include an interpolated prediction block and that can include a block that performs inter-frame prediction from one reference frame, such as a forward prediction block and a backward prediction block, in a picture is called an SH picture. Also, a picture composed of only intra prediction blocks is called an I picture. The pixel value predicted by pixel interpolation is, for example, a pixel average value, a value obtained by weighting and adding pixel values according to a display time difference between the encoding target frame and two reference frames, or a linear prediction formula from pixels of two blocks. The pixel value according to (the linear prediction coefficient is separately stored in the encoded signal) can be used.

The motion estimation means ME0 of FIG. 11 uses the reference frame RefFrms stored in the multi-frame buffer MFrmBuf.
And a picture type PicType indicating the picture type of the encoding target frame are input, motion estimation is performed, and a prediction type Pred indicating a prediction method for blocks such as forward prediction and interpolation prediction
Type0, reference frame number R indicating the selected reference frame
Outputs efNo1 and RefNo2 and motion vectors MV1 and MV2. The motion estimator ME0 is set to 2 when the multiple reference frame interpolation prediction is performed.
Since one reference frame is selected, two reference frame numbers and two motion vectors are output. At this time, the multi-frame buffer MFrmBuf displays the reference frame number RefNo.
Reference blocks RefBlk1 and RefBlk2 corresponding to 1, RefNo2 and motion vectors MV1 and MV2 are output. The pixel interpolating means Pol interpolates the corresponding pixel values of the two reference blocks RefBlk1 and RefBlk2 and outputs the interpolated block RefPol. Since the motion estimation unit ME0 selects one reference frame during inter-frame prediction other than multi-reference frame interpolation prediction,
One reference frame number RefNo1 and one motion vector MV1
Is output. At this time, the multi-frame buffer MFrmBu
f outputs the reference block RefBlk corresponding to the reference frame number RefNo1 and the motion vector MV1. Prediction type PredType0
Indicates interpolation prediction, the switch SW11 is switched to the "1" side and the interpolation block RefPol is used as the predicted image signal Pred. If the prediction type PredType0 indicates an interframe prediction method other than interpolation prediction, the switch SW11 is set to "
The reference block RefBlk is used as the predicted image signal Pred by switching to the 1 "side. Motion vector prediction means MVPred
1 generates a motion vector predictor PMV1 for the motion vector MV1 from the coded motion vector stored in the motion vector buffer MVBuf1. Motion vector subtractor MVSub1
Calculates a difference vector between the motion vector MV1 and the predicted motion vector PMV1 and outputs it as the difference motion vector DMV1. The motion vector MV1 is stored in the motion vector buffer MVBuf1 for generation of a motion vector predictor to be coded thereafter. Similarly, the motion vector predicting means MVPred2 generates a motion vector predictor PMV2, the motion vector subtractor MVSub2 calculates a difference vector between the motion vector MV2 and the motion vector predictor PMV2, and outputs it as a difference motion vector DMV2, which is the motion vector MV2. Store in buffer MVBuf2. I
The picture block header signal generation means GenIHdr generates a block header signal of a block used for an I picture, and SH picture block header signal generation means GenSHH
dr generates a block header signal of a block used for an SH picture, and MH picture block header signal generation means GenMHHdr generates a block header signal of a block used for an MH picture. The method of generating the block header signal will be described later. Switch SW12 has picture type PicType of I
In the case of a picture, it is switched to "0" and the output of the I picture block header signal generation means GenIHdr is output as the block header signal BlkHdr0, and in the case of an SH picture it is "1".
Switching to SH picture block header signal generation means
The output of GenSHHdr is output as a block header signal BlkHdr0, and in the case of an MH picture, the output is switched to "2" and the output of the MH picture block header signal generation unit GenMHHdr is output as a block header signal BlkHdr0. The variable length coding means VLC0 uses the residual coded signal ERes and the reference frame number.
Variable length coding is performed on RefNo1 and RefNo2, differential motion vectors DMV1 and DMV2, block header signal BlkHdr0, and picture type PicType, and the coded signal BitStrm0 is output.

FIG. 13 shows an example of an H.26L macro block. In H.26L, which is currently being standardized by the ITU, a frame is divided into 16x16 size blocks called macroblocks, and coding is performed for each macroblock. Motion compensation can be performed in block units obtained by further dividing a macroblock. In H.26L, motion compensation blocks are hierarchically structured.First, macroblocks are 16x16, 16x8, 8x1.
It is possible to perform motion compensation by dividing it into 6,8x8 blocks. Then only 8x8 blocks, 8x8,8x4,4x8,4
It is possible to perform motion compensation by dividing into x4. The prediction direction of inter-frame reference can be selected in units of 16x16, 16x8, 8x16, 8x8. The macroblock of FIG. 13A is divided into motion compensation blocks of 16 × 8 size, and the prediction direction of each block is forward and bidirectional interpolation.
In the macroblock of FIG. 13 (2), it is first divided into motion compensation blocks of 8x8 size.
The size block is divided into 4x8 size motion compensation blocks. The prediction direction of an 8x8 size block is such that the upper left and lower left blocks are forward, and the upper right and lower right blocks are bidirectional interpolation. In H.26L, the type of prediction, the prediction direction of interframe prediction, and the motion compensation block size of interframe prediction are encoded by a variable length codeword called a block header signal. FIG. 14 is a codeword table of an H.26L block header signal. This codeword table is H.26L Working D
It is a codeword table of the block header signal of the B picture in raft 2.0. The B picture in Working Draft 2.0 of H.26L corresponds to the MH picture described above. The codeword table of the block header signal differs for each picture. The code word table of the block header signal of the I picture and the SH picture (defined as the P picture in H.26L Working Draft 2.0) is not directly related to the present invention, and the description thereof will be omitted. FIG. 14A is a codeword table showing variable-length codewords of the block header signal for the macroblock. Encoding is performed by selecting a row in the codeword table and encoding the corresponding variable-length codeword. The column of "code number" in the table is a value for identifying a variable length code word, and different variable length code words are assigned to different code numbers. The column of "bit length" in the table indicates the bit length of the variable length codeword. The column of "mode" in the table shows the prediction method. 16x16 to 8x8 indicates that the macroblock is coded by interframe prediction and the motion compensation block size is a value of 16x16 to 8x8. Direct indicates that the macroblock is encoded with a special interframe prediction (called direct mode) that uses the motion vector of the encoded frame for interframe prediction.
Intra4x4 and Intra16x16 indicate that the macroblock is in-plane predicted. Since the direct mode and the in-plane prediction are not directly related to the present invention, further description will be omitted. The columns of "first block prediction direction" and "second block prediction direction" in the table indicate the prediction directions of two blocks in a macroblock when the motion compensation block is 16x8, 8x16. When the motion compensation block is 16x16, since there is only one block in the macroblock, only the column of "first block prediction method" is used to indicate the prediction direction of the block. The forward direction in the prediction direction indicates forward prediction, the backward direction indicates backward prediction, and the bidirectional interpolation indicates interpolation prediction. 8x macroblocks
The codeword table of FIG. 14 (B) is shown only in the case of being divided into 8 sizes, that is, only when the variable length codeword of code number 22 of FIG. 4 variable-length codewords are encoded. The column of "bit length" in the table indicates the bit length of the variable length codeword. The column of "mode" in the table shows the prediction method. The column of "Prediction direction" in the table is
Indicates the prediction direction of the block. When the macroblock of FIG. 13 (1) is encoded, the variable length codeword of code number 12 in the codeword table of FIG. 14 (A) is encoded, and the bit amount becomes 7 bits. When the macroblock of FIG. 13 (2) is encoded, first, the code number 22 of the codeword table of FIG.
Variable length codewords are encoded. Next, in the upper left 8x8 block, code number 1 in the codeword table of FIG.
The x8 block is code number 9 in the codeword table of FIG. 14 (B), the lower left 8x8 block is the code number of the codeword table in FIG. 14 (B)
1, the lower right 8x8 block is encoded by the variable length codeword of code number 3 in the codeword table of FIG. 14 (B). FIG. 13 (2) The total bit amount when the macro block is encoded is 27 bits.

FIG. 15 is a conceptual diagram of an image coded signal format of a conventional image coding apparatus. Frame is a coded signal for one frame, Block is a coded signal for one block,
BlkHdr0 indicates a block header signal, RefNo1 and RefNo2 indicate reference frame numbers, and DMV1 and DMV2 indicate differential motion vectors. Blo
If ck is a coded signal of a block by 16x16 size interpolation prediction, reference frame numbers RefNo1, RefNo2 and difference motion vector D for two reference frames in the coded signal.
It will include MV1 and DMV2.

FIG. 16 is a block diagram showing the structure of a conventional image decoding apparatus. The variable length decoding means VLD0 inputs the image coded signal BitStrm0 and performs variable length decoding, and the residual coded signal ERes, differential motion vectors DMV1, DMV2, reference frame numbers RefNo1, RefNo2, block header signal BlkHdr0, picture type PicType Is output. Pixel decoding means ImgDec
Inputs the residual coded signal ERes, and performs the inverse quantization / inverse DCT
Image decoding processing such as conversion is performed, and the residual decoded signal DRes is output. The adder Add adds the residual decoded signal DRes and the predicted image signal Pred, and outputs the decoded image signal DImg to the outside of the image decoding apparatus. I-picture block header analysis means IHd
rDec analyzes the block header signal of the I picture and outputs the prediction type. SH picture block header analysis means SH
HdrDec analyzes the block header signal of the SH picture and outputs the prediction type. MH picture block header analysis means
MHHdrDec analyzes the block header signal of the MH picture and outputs the prediction type. If the picture type PicType indicates an I picture, the switches SW22 and SW23 switch to the "0" side and output the output of the I picture block header analysis means IHdrDec as the prediction type PredType0, and the picture type PicType is SH.
If it shows a picture, it is switched to "1" and the output of the SH picture block header analysis means SHHdrDec is predicted type PredTy
If it is output as pe0 and the picture type PicType indicates an MH picture, it is switched to "2" and the output of the MH picture block header analysis means MHHdrDec is output as the prediction type PredType0. The motion vector predicting means MVPred1 generates a motion vector predictor PMV1 from the decoded motion vector in the motion vector buffer MVBuf1. The motion vector adder MVAdd1 adds the differential motion vector DMV1 and the motion vector predictor PMV1 and outputs the result as a motion vector MV1. The motion vector MV1 is
The motion vector predicting means MVPred2 similarly stores the motion vector buffer MVBuf1 in order to generate a predictive vector at the time of subsequent motion vector decoding.
Motion vector adder MVAdd2 generates a motion vector predictor PMV2 from the decoded motion vector of
V2 and the motion vector predictor PMV2 are added and output as the motion vector MV2, and the motion vector MV2 is added to the motion vector buffer M.
Store in VBuf2. The multi-frame buffer MFrmBuf is
The decoded image signal DImg is stored in a buffer for inter-frame prediction. When the prediction type PredType0 indicates interpolation prediction, the multi-frame buffer MFrmBuf is the motion vector MV1.
And a reference block RefBlk2 corresponding to the motion vector MV2. Pixel interpolation means P
ol interpolates the pixel values corresponding to the two reference blocks RefBlk1 and RefBlk2, and outputs the interpolated block RefPol. When the prediction type PredType0 indicates inter-frame prediction other than interpolation prediction, the multi-frame buffer MFrmBuf outputs the reference frame number RefNo1 and the reference block RefBlk corresponding to the motion vector MV1. If the prediction type PredType0 indicates multiple reference frame interpolation prediction, the switch SW11 is set to "
1 "side, the interpolation block RefPol is used as the prediction image signal Pred. When the prediction type PredType indicates an interframe prediction method other than the multiple reference frame interpolation prediction, the switch SW11 is switched to the" 1 "side, The reference block RefBlk is used as the predicted image signal Pred.
By the processing described above, the image decoding apparatus can generate the image coded signal BitS
The trm0 is decoded and output as the image decoded signal DImg.

[0010]

In the conventional image coding methods MPEG-1, 2, and 4, the MH picture is called a B picture, and the interpolation prediction block has a display time before and after the coding target frame. Interframe prediction was possible only from two reference frames. However, when using B pictures, it is necessary to previously encode the reference frame whose display time is earlier than the encoding target frame, which causes encoding delay and uses B pictures for real-time communication such as videophone. It was difficult to do. Therefore, in H.26L, which is currently being standardized, even when interframe prediction is performed only from reference frames before the encoding target frame, the same interpolation prediction as that of B picture can be used, In both cases, a method of using a reference frame whose display time is earlier than the encoding target frame is being studied. However, when only the reference frame whose display time is earlier than the encoding target frame is used,
When the current H.26L Working Draft 2.0 B-picture block header signal is applied, codewords are assigned to unused backward prediction blocks, which is not optimal in terms of coding efficiency.

[0011]

According to a first aspect of the invention, an input image is divided into rectangular blocks, and two reference frames are selected from a plurality of encoded frames stored in a frame buffer for each block. A predicted image is generated by pixel interpolation, inter-frame prediction is performed from the predicted image, a prediction error is encoded, and a decoded image of a frame that may be used for inter-frame prediction in the encoded frame is stored in the frame buffer. Then, in the image coding method that outputs the image coding signal including the prediction error coding signal and the information that identifies the selected two frames, in the frame that performs the inter-frame prediction by pixel interpolation from the two reference frames. For each block, the information indicating whether the display time of the reference frame is behind or before the display time information of the encoding target frame is coded for both reference frames. Variable length due to different variable length codeword allocation when the display time is before the display time of the target frame and when two reference frames have the display time before and after the display time of the encoding target frame An image encoding method characterized by encoding.

A second aspect of the present invention inputs a coded signal including a prediction error and information for identifying two reference frames,
A reference frame indicated by information for identifying one reference frame is selected from the frame buffer, a predicted image is generated by pixel interpolation, a decoded image is generated from the predicted image and a decoded prediction error, the prediction error is encoded, and a frame is generated. In the image decoding method that stores the decoded image of the frame that may be used for inter-prediction in the frame buffer, outputs the coded signal of the prediction error and the coded signal that contains the information that identifies the selected two frames In the image coding method to
Information that indicates whether the display time of the reference frame is after or before the display time information of the encoding target frame for each block in a frame that performs inter-frame prediction from two reference frames by pixel interpolation Different variable-length codewords are used when both frames have a display time before the display time of the encoding target frame and when two reference frames have a display time before and after the display time of the encoding target frame. This is an image decoding method characterized by variable length decoding by allocation.

According to a third aspect of the invention, an input image is divided into rectangular blocks, two reference frames are selected from a plurality of encoded frames stored in a frame buffer for each block, and a predicted image is obtained by pixel interpolation. Generate, perform inter-frame prediction from the predicted image, encode a prediction error, store a decoded image of a frame that may be used for inter-frame prediction in the encoded frame in a frame buffer, and code the prediction error. In the image coding method for outputting the image coded signal including the coded signal and the information for identifying the selected two frames, when both reference frames have the display time before the display time of the coding target frame Of the two reference frames, the motion vector of the reference frame whose display time is closer to the encoding target frame is set to the encoding target frame and each reference frame. And a prediction vector for a motion vector of a reference frame whose display time is farther, and a difference between the prediction vector and the motion vector of the reference frame whose display time is farther is encoded. This is an image encoding method.

A fourth aspect of the present invention inputs a coded signal including a prediction error and information for identifying two reference frames,
A reference frame indicated by information for identifying one reference frame is selected from the frame buffer, a predicted image is generated by pixel interpolation, a decoded image is generated from the predicted image and a decoded prediction error, the prediction error is encoded, and a frame is generated. In the image decoding method that stores the decoded image of the frame that may be used for inter-prediction in the frame buffer, if both reference frames have the display time earlier than the display time of the encoding target frame, Decode the difference vector to the reference frame whose time is farther, and calculate the motion vector of the reference frame with the closer display time from the coding target frame among the two reference frames and the display time difference between the coding target frame and each reference frame. And a prediction vector for a motion vector of a reference frame whose display time is farther, Vector and display time by adding the difference vector to the more distant the reference frame, the prediction vector and the display time is an image decoding method characterized in that motion vector to the more distant the reference frame.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Specific embodiments of the present invention will be described below with reference to the drawings. (First Embodiment) FIG. 1 is a block diagram of an image coding apparatus according to the first embodiment. Units that perform the same operations as those in the block diagram showing the configuration of the conventional image coding apparatus in FIG. 11 and signals of the same operations are denoted by the same symbols, and description thereof is omitted. The motion estimation means ME inputs a reference frame RefFrms, a picture type PicType, and a prediction direction PDir at the time of multiple reference frame interpolation prediction, and performs motion estimation. The prediction direction is used only when encoding MH pictures, and is forward when the display times of both reference frames are before the target frame for encoding, and when the display time of the reference frame is before and after the target frame for encoding. Indicates bidirectional.

FIG. 2 is a conceptual diagram of a forward prediction MH picture. Frm is an encoding target frame whose picture type is MH picture. FwRef1 and FwRef2 are target frames Frm
An encoded frame whose display time is earlier is shown. block
Blk1 is a reference block RefBlk included in the frame FwRef2.
11 is a forward-interpolation prediction block that is inter-frame predicted by pixel interpolation from 11 and a reference block RefBlk12 included in the frame FwRef1. The block Blk2 is a forward prediction block inter-frame predicted from the reference block RefBlk2 included in the frame FwRef1.

FIG. 3 shows an example of the macroblock according to the first embodiment. The macroblock of FIG. 3 (1) is divided into motion compensation blocks of 16x8 size, and the prediction direction of each block is forward and forward interpolation. In the macroblock of FIG. 13 (2), first, it is divided into 8x8 size motion compensation blocks, and the upper right 8x8 size block is further divided into 4x8 size motion compensation blocks. The prediction direction of an 8x8 size block is such that the upper left and lower left blocks are forward, and the upper right and lower right blocks are forward interpolation. FIG. 4 is a codeword table of the block header signal of the first embodiment, which is an example of the codeword table of the block header signal in the MH picture of the present embodiment. This codeword table removes the "backward" prediction, which is not used when only the reference frame before the encoding target frame is used, from the codeword table of Fig. 14, and the "bidirectional interpolation" prediction is performed before the encoding target frame. of 2
This is changed to "forward interpolation" prediction, which indicates that interpolation prediction is performed from one reference frame. Efficient coding is possible because unused codes are removed. Figure 3
When the macroblock of (1) is coded, it is shown in Fig. 4 (A).
The variable-length codeword of code number 6 in the codeword table is coded, and the bit amount becomes 5 bits. When the macroblock of FIG. 3 (2) is encoded, first, the variable length codeword of code number 12 in the codeword table of FIG. 4 (A) is encoded. next,
The upper left 8x8 block is code number 1 in the codeword table of FIG. 4 (B), the upper right 8x8 block is code number 6 in the codeword table of FIG. 4 (B), and the lower left 8x8 block is of the code word table of FIG. 4 (B). The code number 1 in the code word table and the lower right 8x8 block are encoded by the variable length code word of the code number 2 in the code word table of FIG. 4 (B). The total bit amount when the macroblock in FIG. 3 (2) is encoded is 21
Is a bit. In either case of FIGS. 4 (A) and 4 (B), the required bit amount is smaller than in the case of simply replacing “bidirectional interpolation” with “forward interpolation” in the codeword table of FIG. Also, in many cases of possible combinations of the mode, the first block prediction direction, and the second block prediction direction, more necessary bits than in the case of replacing "bidirectional interpolation" with "forward interpolation" in the codeword table of FIG. You can see that the quantity is small. FIG.
GenFMHHdr is a block header signal generation unit GenFMHHdr for forward prediction MH picture used when generating a block header when the display time of the reference frame is always before the display time of the encoding target frame. Forward prediction MH picture block header signal generation means GenFMHHdr is a prediction type PredT
A block header is generated from ype by the codeword table shown in FIG. 4, for example. The bidirectionally predicted MH picture block header signal generation unit GenBMHHdr is used during bidirectional prediction in which prediction is performed from reference frames before and after the display time of the frame to be coded, and shows the configuration of the conventional image coding apparatus in FIG. The block header signal generating means for MH picture in the block diagram GenMHHdr has the same operation. Switch SW
13 is "if the picture type PicType indicates an I picture."
If the output of the block header signal generating unit GenIHdr for I picture is switched to 0 "and the block header signal BlkHdr is output, and the picture type PicType indicates SH picture"
Switch to 1 "side Output the output of the SH picture block header signal generation unit GenSHHdr as a block header signal BlkHdr. If the picture type PicType indicates MH picture and the prediction direction PDir indicates bidirectional, switch to" 2 "side. Block header signal generation means for directional prediction MH pictures GenBMHHdr
Is output as a block header signal BlkHdr, and if the picture type PicType indicates MH picture and the prediction direction PDir indicates forward, it is switched to "3" and the output of the block header signal generation unit GenFMHHdr for forward prediction MH picture Output as signal BlkHdr. Variable length coding means
VLC is the residual coded signal ERes, the reference frame number RefNo1,
The RefNo2, the differential motion vectors DMV1 and DMV2, the block header signal BlkHdr, the picture type PicType, and the prediction direction PDir are subjected to variable length coding and output as a coded signal BitStrm.

FIG. 5 shows an image coded signal format according to the first embodiment. The same symbols are attached to the same signals as in the conceptual diagram of the image coded signal format of the conventional image coding apparatus in FIG. 15, and the description is omitted. The difference from the image coded signal format of the conventional image coding apparatus shown in FIG. 15 is that the prediction direction PDir is included for each frame and the block header signal BlkHdr based on a new code word table is included. As described above, according to the present embodiment, it is not necessary to encode one of the reference frame numbers, so that the encoding efficiency can be improved.

In this embodiment, the prediction direction PDir is coded in the coded signal, but the image decoding apparatus can distinguish the forward direction and the bidirectional direction from the coding order of the display time of each frame. Therefore, the prediction direction PDir is not necessary. For example, if the display times arranged in the encoding order are in ascending order, the prediction direction is known to be forward. Otherwise, the prediction direction is known to be bidirectional.

In the present embodiment, the display times of the reference frames during the multiple reference frame interpolation prediction are both before the frame to be coded, but the multiple reference frame interpolation prediction is performed by the same method as the present invention. Both of the display times of the reference frames at the time can be processed even after the encoding target frame. In this case, the motion estimation means ME performs motion estimation from a frame after the frame to be encoded, and "forward" to "rear" and "forward interpolation" to the frame to be encoded in the codeword table in FIG. The block header signal may be encoded by the code word table in which the forward prediction MH picture block header signal generation unit GenFMHHdr performs the above replacement by replacing the backward prediction frame with the later display time with interpolation prediction. .

(Second Embodiment) FIG. 6 is a block diagram of an image decoding apparatus according to the second embodiment. Units performing the same operations and signals of the same operations as those in the block diagram showing the configuration of the conventional image decoding apparatus in FIG. 16 are denoted by the same symbols, and description thereof will be omitted.

The variable length decoding means VLD uses the coded image signal Bit
Strm is input, variable length decoding is performed, and residual coded signal ERe
s, reference frame numbers RefNo1, RefNo2, difference motion vector
DMV1, DMV2, block header signal BlkHdr, picture type P
Output icType and prediction direction PDir. FMHHdrDec is a forward prediction MH picture block header analysis unit FMHHdrDec used for block header analysis when the display time of the reference frame is always before the display time of the encoding target frame. Bidirectional prediction MH picture block header analysis means BM
HHdrDec is used at the time of bidirectional prediction in which prediction is performed from reference frames before and after the display time of the encoding target frame, and MH of the block diagram showing the configuration of the conventional image decoding device of FIG.
This is a means for performing the same operation as the picture block header analysis means MHHdrDec. The switches SW24 and SW25 switch to the "0" side when the picture type PicType indicates an I picture, and outputs the output of the I picture block header analysis means IHdrDec as the prediction type PredType, and the picture type PicType is SH.
If it shows a picture, it is switched to "1" and the output of the SH picture block header analysis means SHHdrDec is predicted type PredTy
If it is output as pe, and the picture type PicType indicates MH picture and the prediction direction PDir indicates bidirectional, it is switched to "2" side and the block header analyzing means BMHH for bidirectional predicted MH picture is output.
If the output of drDec is output as the prediction type PredType, and the picture type PicType indicates MH picture and the prediction direction PDir indicates bidirectional, it is switched to "3" side and the output of the forward prediction MH picture block header analysis means FMHHdrDec is predicted type. Pre
Output as dType.

As described above, according to the present embodiment, the image coded signal coded by the block diagram of the image coding apparatus using the image coding method of the present invention of FIG. 1 described in the first embodiment. Can decrypt BitStrm correctly.

Although the prediction direction PDir is coded in the coded signal in the present embodiment, the prediction direction PDir
Even when ir is not present in the encoded signal, the image decoding device can distinguish between forward and bidirectional based on the encoding order of the display time of each frame, and thus the prediction direction PDir is not necessary. For example, if the display times arranged in encoding order are ascending order,
It can be seen that the prediction direction is forward. Otherwise, the prediction direction is known to be bidirectional.

In the present embodiment, the display times of the reference frames during the multiple reference frame interpolation prediction are both before the encoding target frame, but the multiple reference frame interpolation prediction is performed by the same method as the present invention. Both of the display times of the reference frames at the time can be processed even after the encoding target frame. In this case, "forward" in the code word table of FIG. 4 is replaced with "backward", and "forward interpolation" is replaced with "backward interpolation" for performing interpolation prediction from a reference frame whose display time is later than the encoding target frame, The block header signal may be decoded by the code word table that has been replaced by the block header analysis unit FMHHdrDec for forward prediction MH pictures.

(Third Embodiment) FIG. 7 is a block diagram of an image coding apparatus according to the third embodiment. Units that perform the same operations as those in the block diagram of the image coding apparatus according to the first embodiment in FIG. 1 and signals of the same operations are denoted by the same symbols, and description thereof will be omitted.

In the MH picture in which the inter-frame prediction is performed from the reference frame whose display time is earlier than the encoding target frame, since the two reference frames at the time of interpolation prediction are the frames of different display times, the encoding target frame is displayed. It can be said that the reference frame farther from the time is not at least the frame immediately before the encoding target frame. In an MH picture in which inter-frame prediction is performed from a reference frame whose display time is earlier than the encoding target frame, most blocks select a forward prediction block that refers to a frame having a display time immediately before the encoding target frame. . Therefore, in the motion vector prediction method of H.26L that generates a prediction vector by using only the motion vector that refers to the same frame in the motion vectors of the peripheral blocks, a reference frame farther from the display time of the encoding target frame is Since there are almost no motion vectors that refer to the same frame in the peripheral blocks, the motion vector prediction efficiency deteriorates. So MH
When performing inter-frame prediction from a reference frame whose display time is earlier than the encoding target frame in a picture, a vector of a reference frame farther from the display time of the encoding target frame is encoded by the method described below.

FIG. 8 is an explanatory diagram of the motion vector predictor according to the third embodiment. Frm is the frame to be coded, Ref1 is a reference frame that has a display time that is close to the frame to be coded Frm in the two reference frames, and Ref2 is a reference frame that is farther from the frame to be coded Frm in the two reference frames. Indicates a frame. Blk is a block to be encoded, RefBlk1 is a reference block by motion compensation in the reference frame Ref1, RefB
lk2 is a reference block by motion compensation in the reference frame Ref2. MV1 is a motion vector for the reference block RefBlk1, MV2 is a motion vector for the reference block RefBlk2,
TRD1 is the display time difference between the encoding target frame Frm and the reference frame Ref1, TRD2 is the display time difference between the encoding target frame Frm and the reference frame Ref2, and SMV1 is TRD2 / TR in the motion vector MV1.
The motion vector scaled by multiplying D1 and MVD2 indicate the difference vector between the motion vector MV2 and the vector SMV1. Assuming that the direction and speed of movement of the object in the screen are constant between the encoding target frame Frm, the reference frame Ref1, and the reference frame Ref2, the difference vector MVD2 becomes close to 0, and this difference vector MVD2 has a small absolute value. By performing variable length coding such that the word length becomes shorter, efficient coding is possible.

Motion vector scaling means MVScaler
Performs SMV1 = (TRD2 / TRD1) × MV1 scaling processing as described with reference to FIG. When the block is interpolated and predicted from two reference frames whose display time is earlier than the encoding target, switch SW26 to the "1" side
Output SMV1 as a prediction vector. When the block is interpolated and predicted from two reference frames having display times before and after the encoding target, the switch SW26 is switched to the "0" side, and the vector PMV2 is output as the prediction vector.
The variable length coding means VLC2 has a residual coded signal ERes, reference frame numbers RefNo1, RefNo2, and differential motion vectors DMV1, DMV.
2, block header signal BlkHdr, picture type PicType,
The prediction direction PDir is variable length coded and output as a coded signal BitStrm2.

The image coding format of the image coding apparatus according to this embodiment is the same as the image coding signal format shown in FIG. 5, and the scaling of the motion vector MV1 shown in this embodiment is used as the difference vector DMV2. The result of the differential encoding that uses the result as the prediction vector will be stored.

(Fourth Embodiment) FIG. 9 is a block diagram of an image decoding apparatus according to the fourth embodiment. Units that perform the same operations as those in the block diagram of the image decoding apparatus according to the second embodiment in FIG. 6 and signals that have the same operations are denoted by the same symbols, and description thereof will be omitted.

The variable length decoding means VLD2 is used for the image coded signal Bi.
Input tStrm2, perform variable length decoding, and residual coded signal ER
es, reference frame numbers RefNo1 and RefNo2, differential motion vectors DMV1 and DMV2, block header signal BlkHdr, picture type PicType, and prediction direction PDir are output. The motion vector scaling means MVScaler uses the SMV1 as described in FIG.
= (TRD2 / TRD1) × MV1 scaling processing is performed. When the block is predicted to be interpolated from the two reference frames whose display time is earlier than the encoding target, set the switch SW14 to "1".
Switch to the side and output the vector SMV1 as a prediction vector. 2 blocks have display times before and after encoding
When the interpolation prediction is performed from the reference frames, the switch SW14 is switched to the "0" side, and the vector PMV2 is output as the prediction vector.

As described above, according to the present embodiment, the image coded signal coded by the block diagram of the image coding apparatus using the image coding method of the present invention of FIG. 7 described in the third embodiment. BitStrm2 can be decrypted correctly.

(Embodiment 7) Further, a program for realizing the configuration of the image coding method or the image decoding method shown in each of the above embodiments is recorded in a storage medium such as a flexible disk. As a result, the processing shown in each of the above embodiments can be easily implemented in an independent computer system.

FIG. 10 is an explanatory diagram of a storage medium for storing a program for realizing the image coding method and the image decoding method of the first to fourth embodiments by a computer system.

FIG. 10B shows the appearance, sectional structure, and flexible disk of the flexible disk as viewed from the front, and FIG. 10A shows an example of the physical format of the flexible disk, which is the recording medium body. .
The flexible disk FD is built in a case F, and a plurality of tracks Tr are formed concentrically from the outer circumference to the inner circumference on the surface of the flexible disk FD, and each track is angularly divided into 16 sectors Se. ing. Therefore, in the flexible disk storing the program, the image encoding method as the program is recorded in the area allocated on the flexible disk FD.

FIG. 10C shows a structure for recording / reproducing the program on / from the flexible disk FD. When the program is recorded on the flexible disk FD, the image encoding method or the image decoding method as the program is written from the computer system Cs via the flexible disk drive FDD. Further, when the image coding method is constructed in the computer system by the program in the flexible disk, the program is read from the flexible disk by the flexible disk drive and transferred to the computer system.

In the above description, the flexible disk is used as the recording medium, but the same can be applied by using the optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card and a ROM cassette that can record the program can be similarly used.

[0039]

As described above in detail, according to the image coding method and the image decoding method of the present invention, the display time of the coding target frame is predicted by the inter-frame prediction from the previous two reference frames. In this case, efficient coding of the block header signal and the motion vector can be realized.

[Brief description of drawings]

FIG. 1 is a block diagram of an image coding apparatus according to a first embodiment.

[Fig. 2] Conceptual diagram of forward prediction MH picture

FIG. 3 is an example of a macroblock according to the first embodiment.

FIG. 4 is a codeword table of a block header signal according to the first embodiment.

FIG. 5 is an image coded signal format according to the first embodiment.

FIG. 6 is a block diagram of an image decoding device according to a second embodiment.

FIG. 7 is a block diagram of an image encoding device according to a third embodiment.

FIG. 8 is an explanatory diagram of a motion vector predictor according to the third embodiment.

FIG. 9 is a block diagram of an image decoding device according to a fourth embodiment.

FIG. 10 is an explanatory diagram of a storage medium for storing a program for realizing the image encoding method and the image decoding method of the first to fourth embodiments by a computer system.

FIG. 11 is a block diagram showing a configuration of a conventional image encoding device.

[Figure 12] Conceptual diagram of MH picture

[Fig. 13] Example of H.26L macroblock

FIG. 14 is a codeword table of H.26L block header signals.

FIG. 15 is a conceptual diagram of an image coded signal format of a conventional image coding apparatus.

FIG. 16 is a block diagram showing the configuration of a conventional image decoding device.

[Explanation of symbols]

ImgEnc Image coding means ImgDec Image decoding means Add Adder Sub Subtractor MFrmBuf Multi-frame buffer ME, ME0 Motion estimation means MVPred1, MVPred2 Motion vector prediction means MVBuf1, MVBuf2 Motion vector buffers VLC, VLC0, VLC2 Variable length coding means VLD , VLD0, VLD2 Variable length decoding means Pol Pixel interpolation means GenFMHHdr Forward prediction MH picture block header signal generation means GenBMHHdr Bidirectional prediction MH picture block header signal generation means GenMHHdr MH picture block header signal generation means GenSHHdr SH picture block header Signal generation means GenIHdr I picture block header Signal generation means FMHHdrDec Forward prediction MH picture block header analysis means BMHHdrDec Bidirectional prediction MH picture block header analysis means MHHdrDec MH picture block header analysis means SHHdrDec SH picture block header analysis means IHdrDec Block header solution for I-picture Means MVAdd1, MVAdd2 vector summer MVSub1, MVSub2 vector subtracter MVPred1, MVPred2 motion vector prediction means MVScaler motion vector scaling unit SW11~SW14, .SW21~SW26 switch Cs computer system FD flexible disk FDD flexible disk drive

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Toshishi Kondo             1006 Kadoma, Kadoma-shi, Osaka Matsushita Electric             Sangyo Co., Ltd. (72) Inventor Kiyoshi Abe             1006 Kadoma, Kadoma-shi, Osaka Matsushita Electric             Sangyo Co., Ltd. F-term (reference) 5C059 MA00 MA05 MA12 MC38 ME01                       NN11 NN16 NN21 PP04 PP05                       PP06 PP07 RB09 RC11 RC16                       RC40 SS07 SS20 TA57 TA58                       TA61 TB04 TC00 TC12 TC42                       TD05 TD11 UA02 UA05 UA32                 5J064 AA02 BA09 BA13 BB03 BB04                       BC01 BC08 BC14 BC29 BD01

Claims (10)

[Claims] 1. An input image is divided into rectangular blocks, two reference frames are selected from a plurality of encoded frames stored in a frame buffer for each block, and a predicted image is generated by pixel interpolation, Performs inter-frame prediction from the predicted image, encodes the prediction error, stores the decoded image of the frame that may be used for inter-frame prediction in the encoded frame in the frame buffer, and selects the encoded signal of the prediction error In the image coding method for outputting the image coded signal including the information for identifying the two frames,
Information that indicates whether the display time of the reference frame is before or after the display time information of the encoding target frame for each block in a frame that performs inter-frame prediction from pixel reference frames by pixel interpolation. Also, different variable-length codewords are assigned depending on whether the display time is before the display time of the encoding target frame or when the two reference frames have the display time before and after the display time of the encoding target frame. An image coding method characterized in that variable length coding is performed according to the above. 2. A coded signal including a prediction error and information for identifying two reference frames is input, a reference frame indicated by the information for identifying the two reference frames is selected from a frame buffer, and pixel interpolation is performed. A predicted image is generated, and a decoded image is generated from the predicted image and a predicted error that is decoded,
In the image decoding method that encodes the prediction error and stores the decoded image of the frame that may be used for inter-frame prediction in the frame buffer, the prediction error coded signal and the information that identifies the selected two frames In an image coding method that outputs a coded image signal that includes a frame that performs inter-frame prediction from two reference frames by pixel interpolation, for each block, display the reference frame for the display time information of the encoding target frame Information indicating whether the time is after or before is displayed when both reference frames have a display time before the display time of the encoding target frame and when the two reference frames are before the display time of the encoding target frame. An image decoding method characterized in that variable length decoding is performed by assigning different variable length code words when the display time is later. 3. An input image is divided into rectangular blocks, two reference frames are selected from a plurality of encoded frames stored in a frame buffer for each block, and a predicted image is generated by pixel interpolation, Performs inter-frame prediction from the predicted image, encodes the prediction error, stores the decoded image of the frame that may be used for inter-frame prediction in the encoded frame in the frame buffer, and selects the encoded signal of the prediction error In the image coding method for outputting the image coded signal including the information for identifying the two frames,
If both reference frames have a display time that is earlier than the display time of the encoding target frame, the motion vector of the reference frame that is closer in display time than the encoding target frame among the two reference frames is the encoding target. Scaling is performed by the ratio of the display time difference between the frame and each reference frame to form a prediction vector for the motion vector of the reference frame whose display time is farther, and the difference between the prediction vector and the motion vector of the reference frame whose display time is farther is encoded. An image coding method characterized by the above. 4. A coded signal including a prediction error and information for identifying two reference frames is input, a reference frame indicated by the information for identifying the two reference frames is selected from a frame buffer, and pixel interpolation is performed. A predicted image is generated, and a decoded image is generated from the predicted image and a predicted error that is decoded,
In the image decoding method that encodes the prediction error and stores the decoded image of the frame that may be used for inter-frame prediction in the frame buffer, both reference frames have a display time that is earlier than the display time of the encoding target frame. , The difference vector to the reference frame having the farther display time is decoded, and the motion vector of the reference frame having the closer display time from the coding target frame among the two reference frames is set as the coding target frame. As a prediction vector for the motion vector of the reference frame whose display time is farther by scaling at the ratio of the display time difference to each reference frame, the difference vector to the reference frame whose display time is farther is added, and the prediction vector is Image decoding method characterized by using a motion vector to a reference frame whose display time is further away 5. A difference device for inputting an image signal, performing a difference between the image signal and a predicted image, and outputting the difference signal as a residual signal,
An image encoding unit that performs an image encoding process on the difference signal and outputs it as a residual encoded signal; an image decoding unit that decodes the residual encoded signal and outputs it as a residual decoded signal;
Motion estimation using an adder that outputs the reconstructed image by adding the residual decoded signal and the predicted image, a multi-frame buffer that stores the reconstructed image, and a reference frame commonly used for a plurality of blocks Of the reference frame and the motion vector that can be selected in block units, and the two reference frames detected by the motion estimation unit and the two reference blocks referred to by the motion vector corresponding to each reference frame. Pixel interpolating means for performing pixel interpolation and outputting as a predicted image, and forward prediction MH for coding a block header signal of a frame for which interpolative prediction is performed from two reference frames having a display time before the target frame for encoding.
A block header generating unit for a picture, and a bidirectional prediction MH picture for encoding a block header signal of a frame for which interpolation prediction is performed from a reference frame having a display time before the encoding target frame and a reference frame having a display time after the encoding target frame A block header generating means, a residual coded signal, a block header signal, a motion vector, a reference frame number for identifying a reference frame, and two reference frames at the time of interpolation prediction, both before the display time of the target frame for encoding. An image coding apparatus comprising variable length coding means for variable length coding a prediction direction signal indicating whether it is before or after and outputting it as a coded signal. 6. A variable length decoding means for inputting an image coded signal, performing variable length decoding, and separating it into a residual coded signal, a motion vector, a block header signal, a reference frame number and a prediction direction signal, and the residual. An image decoding unit that decodes a coded signal and outputs a decoded residual signal, an adder that adds the residual decoded signal and a predicted image signal and outputs a decoded image, a multi-frame buffer that stores the decoded image, Pixel interpolation means for performing pixel interpolation of two reference frames indicated by the information for identifying the reference frame and two reference blocks referred to by the motion vector corresponding to each reference frame, and outputting as a prediction image, and the prediction direction signal A block header for a forward prediction MH picture that analyzes a block header signal when both reference frames have a display time earlier than the encoding target frame; Image decoding characterized in that the prediction direction signal includes a bidirectionally predicted MH picture block header that analyzes the block header signal when two reference frames have display times before and after the frame to be encoded apparatus. 7. A storage medium storing a program for performing image encoding by a computer, the program dividing the input image into rectangular blocks and storing the blocks in a frame buffer for each block. Two reference frames are selected from a plurality of encoded frames, a prediction image is generated by pixel interpolation, inter-frame prediction is performed from the prediction image, a prediction error is encoded, and inter-frame prediction is performed using the encoded frame. A decoded image of a frame that may be used is stored in a frame buffer, and an image coding method for outputting a coded signal of a prediction error and an image coded signal including information for identifying the selected two frames, When displaying the coding target frame for each block in a frame that performs inter-frame prediction by pixel interpolation from two reference frames The information indicating whether the display time of the reference frame is later or earlier than the information is displayed when both the reference frames have the display time before the display time of the encoding target frame, and when the two reference frames are the encoding target frames. A storage medium for performing variable-length coding by assigning different variable-length codewords in the case of having a display time before and a display time after. 8. A storage medium storing a program for performing image decoding by a computer, wherein the program inputs a coded signal including a prediction error and information for identifying two reference frames to the computer. , A reference frame indicated by information for identifying the two reference frames is selected from a frame buffer, a predicted image is generated by pixel interpolation, a decoded image is generated from the predicted image and the decoded prediction error, and the prediction error is encoded. In the image decoding method that stores the decoded image of the frame that may be used for inter-frame prediction in the frame buffer, the image coding that includes the coded signal of the prediction error and the information that identifies the selected two frames In the image coding method for outputting a signal, 2
Information that indicates whether the display time of the reference frame is before or after the display time information of the encoding target frame for each block in a frame that performs inter-frame prediction from pixel reference frames by pixel interpolation. Also, different variable-length codewords are assigned depending on whether the display time is before the display time of the encoding target frame or when the two reference frames have the display time before and after the display time of the encoding target frame. A storage medium characterized in that variable-length decoding is performed by. 9. A storage medium storing a program for performing image encoding by a computer, the program dividing the input image into rectangular blocks, and storing the blocks in a frame buffer for each block. Two reference frames are selected from a plurality of encoded frames, a prediction image is generated by pixel interpolation, inter-frame prediction is performed from the prediction image, a prediction error is encoded, and inter-frame prediction is performed using the encoded frame. A decoded image of a frame that may be used is stored in a frame buffer, and an image coding method for outputting a coded signal of a prediction error and an image coded signal including information for identifying the selected two frames, If both reference frames have a display time that is earlier than the display time of the encoding target frame, encoding is performed within the two reference frames. The motion vector of the reference frame whose display time is closer to the target frame is the prediction vector for the motion vector of the reference frame whose display time is farther by scaling the ratio of the display time difference to the coding target frame and each reference frame, and the prediction vector A storage medium characterized by causing a difference from a motion vector of a reference frame whose display time is farther to be encoded. 10. A storage medium storing a program for performing image decoding by a computer, wherein the program inputs a coded signal including a prediction error and information for identifying two reference frames to the computer. , Above 2
A reference frame indicated by information for identifying one reference frame is selected from the frame buffer, a predicted image is generated by pixel interpolation, a decoded image is generated from the predicted image and a decoded prediction error, the prediction error is encoded, and a frame is generated. In the image decoding method that stores the decoded image of the frame that may be used for inter-prediction in the frame buffer, if both reference frames have the display time earlier than the display time of the encoding target frame, Decode the difference vector to the reference frame whose time is farther, and calculate the motion vector of the reference frame with the closer display time from the coding target frame among the two reference frames and the display time difference between the coding target frame and each reference frame. And a prediction vector for a motion vector of a reference frame whose display time is farther, Vector and display time by adding the difference vector to the more distant the reference frame, a storage medium, wherein the prediction vector and the display time in which to perform the motion vector to the more distant the reference frame.