JP6513847B2 - Video coding method - Google Patents

Description

  The present invention relates to encoding / decoding techniques for compressed moving image data, and more particularly to techniques for encoding / decoding in block units.

  An outline of a moving image encoding / decoding method for performing encoding / decoding processing in block units will be described. As shown in FIG. 3, one frame of a moving image is composed of one luminance signal (Y signal: 61) and two color difference signals (Cr signal: 62, Cb signal: 63). The image size of the signal is 1/2 of the luminance signal in both vertical and horizontal directions. In a general video standard, each frame of a moving image is divided into small blocks as shown in FIG. 3, and reproduction processing is performed in block units called macroblocks. FIG. 5 shows the structure of the macroblock. The macro block is composed of one Y signal block 30 of 16 × 16 pixels, and a Cr signal block 31 and a Cb signal block 31 of 8 × 8 pixels which spatially coincide with each other.

  Video encoding is processed on a macroblock basis as indicated above. The encoding method is roughly divided into two types, which are called intra encoding (intra mode) and predictive encoding (inter mode). Intra coding is an error macroblock obtained by taking a difference between a predicted macroblock image generated by performing spatial prediction on an input macroblock image to be encoded or on the input macroblock image and the input macroblock image. It is a data compression method in the spatial direction that applies DCT to an image and quantizes and encodes each transform coefficient. This intra coding is arranged in a macro block (also including the first coded frame) having no part similar to the previous frame, a part where it is desired to eliminate accumulated operation error accompanying DCT, or the like.

  The algorithm of predictive coding is called MC-DCT (Motion Compensated-Discrete Cosine Transform). Motion compensation is a time-direction compression technique that searches for a portion similar to the content of the target macroblock from the reference frame and encodes the amount of motion (motion vector). Usually, a macroblock is further divided into small blocks, and a motion vector is calculated for each small block. For example, in MPEG-4 Part 10 (Advanced Video Coding), a macroblock division type (luminance component) at the time of motion compensation is considered as shown in FIG. The basics are four types of type 51 to type 54. In the case of the type 54, the structure is such that each of the four 8 × 8 blocks 54-0 to 54-3 is further selected from the types 54a, 54b, 54c, 54d and five types of intra coding. As a method of detecting a motion vector in each small block, a portion in which the sum of absolute values or the sum of squared errors of prediction error signals in the block is small is selected. When the calculation speed is important, the sum of absolute values is used, and when the coding efficiency is sought, the sum of squared errors is used. Furthermore, in the case of pursuing coding efficiency, the method of converting the code amount into the evaluation value of the sum of squared errors and calculating the optimal coding mode and motion amount using both the prediction error and the code amount is applied. Sometimes.

  FIG. 4 shows the processing structure of motion compensation in one block. FIG. 4 is a diagram showing the prediction block 75 and the motion vector 76 on the previous frame 73 (reference frame) for the luminance signal block 72 of the current frame 71 surrounded by a thick frame. The motion vector 76 indicates the movement from the block 74 (broken line) on the previous frame corresponding to the same position spatially with respect to the thick frame block of the current frame to the region of the predicted block 75 on the previous frame. (The motion vector length for the color difference signal is half that of the luminance signal and is not encoded.) After this motion compensation, DCT is performed on the error macroblock image obtained by taking the difference between the prediction macroblock image composed of a plurality of prediction blocks and the input macroblock image, and the transform coefficients are quantized and coded. Turn Furthermore, the motion vector in the detected macroblock is also encoded. Since motion vectors between adjacent blocks have close values, normally, a difference value between the motion vector of the adjacent block and that of the adjacent block is encoded.

  As a motion compensation method for predictive coding, bidirectional prediction in which MC is performed using temporally past and future frames as a reference frame in addition to forward predictive coding that performs MC using temporally past frames as a reference frame There is coding. In the motion compensation of the forward prediction coding described above, only forward prediction is dealt with, but in the motion compensation of bidirectional coding, in addition to Forward Prediction, Backward Prediction, Forward Prediction Block And bi-directional prediction that interpolates each pixel in the backward prediction block to generate an interpolated prediction block, as well as using a motion vector from a temporally future frame to a past frame Handle direct prediction (Direct) that performs bi-directional prediction. In forward prediction, backward prediction, and bidirectional prediction modes, motion vectors corresponding to forward motion vectors, motion vectors corresponding to backward motion vectors, motion vectors corresponding to forward motion vectors and backward motion vectors , But does not require motion vector coding in this Direct mode. FIG. 9 shows the prediction concept of the Direct mode. As shown in the figure, first, the forward motion vector 132 of the block (131, collocated-block) on the backward reference frame 130 at the same spatial position as the prediction target block 121 on the current frame 120 is temporally The forward motion vector 122 and the backward motion vector 123 are converted at a ratio of the frame position. Then, interpolation processing similar to bidirectional prediction is performed using these converted motion vectors.

  A frame in which intra coding is applied to all macroblocks is configured by I-picture, a frame configured by forward prediction coding or intra coding is configured by P-picture, bidirectional coding or intra coding The frame to be sent is called B-picture.

  So far, general encoding / decoding methods have been described, but in recent encoding / decoding methods, the function of expanding the degree of freedom of selection is applied. Here are some of the new features. These functions are also considered in MPEG-4 Part 10 (Advanced Video Coding).

1. Multiple Reference Frames In the above, the number of reference frames used for motion compensation is one in P-picture, and two in B-picture: forward reference picture and backward reference picture. However, there is a method of preparing a plurality of frames as reference frames in the past direction and in the future direction, and selecting different reference frames in units of macro blocks or small blocks obtained by dividing macro blocks. Also, conventionally, the reference frame is I-picture or P-picture, but it is also possible to select B-picture as a reference picture.

2.2 Direction Reference Frame Prediction It is possible to include temporally past frames as candidates for backward reference picture in multiple reference frames. In this method, it is also permitted that all backward reference pictures are temporally past frames. Therefore, the term Bi-predictive is used in place of Bi-directional as a generic term. When the two reference frames 140 and 150 are both temporally past frames or both future frames, the coding method of the motion vector 127 for the reference frame 150 far from the current frame is changed. As shown in FIG. 10, a difference vector 126 between the motion vector 125 and the motion vector 127 obtained by converting the motion vector 124 with respect to the reference frame 140 close to the current frame 121 at a temporal frame position ratio Be

3. Changing the Coding / Decoding Order Conventionally, the processing order of each frame is that I-picture and P-picture are in display order, and consecutive B-pictures located between two I / P-pictures in time are The format of FIG. 11 follows the latter I / P-picture immediately after. However, the new function is not limited to this processing order as long as the display delay range is allowed. Also, when using the concept of Bi-predictive, B-picture may occur even if it does not have a reference frame for backward prediction. Note that the display order described above is encoded as data header information of video data, or corresponds to a superordinate concept of video data, such as a communication layer that performs synchronous processing of video data and audio / audio data, and division and delivery of data. Since the file format is managed, there is no problem of display displacement due to the change of encoding / decoding processing order.

4. Frame Identification Conventionally, information indicating a display position has been encoded for each frame. However, this display position information may not match the time information included in the communication packet or file format that is actually applied at the time of display. In order to avoid this problem, in video data, a method of managing each frame by processing number only is also considered.

  However, in the video encoding / decoding method introduced a new function, backward reference set by default when there is no backward reference frame to be used for the direct mode or from a plurality of backward reference frames It is conceivable that the frame is not a future frame. In such a frame, direct mode can not be applied. In addition, when management of each frame is managed by the number in decoding order, it can not be determined whether or not the backward reference frame is available. When B-picture is selected as a backward reference frame to be used in the direct mode, it may be considered that the collocated block does not have a forward motion vector. In such a block, direct mode can not be applied.

  In view of the above problems, an object of the present invention is to provide an encoding / decoding method that can apply the direct mode efficiently.

  The decoder is provided with information indicating whether a backward reference frame set by default is available for the direct mode. It provides a switching procedure to a price method and a price method that can be applied when the collocated block has no available forward motion vector.

  Further, the present invention relates to a moving image encoding and decoding method which receives information on a motion vector, performs motion compensation using the recorded reference image and the information on the motion vector, and synthesizes a predicted image. A motion compensation has a plurality of block modes including a mode in which the motion vector does not accompany decoding, selects a prediction mode representing a prediction direction, and makes reference to a frame to be referred in prediction of each prediction direction in the prediction mode A method of selecting from candidates and selecting motion vector information to be used in the prediction mode. In particular, the prediction mode is performed based on whether or not a block adjacent to the current block has a motion vector.

  Also, the frame to be referred is one for selecting one reference frame from the plurality of reference frames identified by the index number, and when the plurality of adjacent blocks apply the prediction of the prediction direction. Selects a reference frame to be used by any of the adjacent blocks, and when only one of the adjacent blocks applies the prediction of the prediction direction, a reference to the index number applied by the one adjacent block It is disclosed to select a frame and to select a reference frame with index number 0 if the adjacent block does not adopt the selected prediction mode.

  Also disclosed is a method of including, in a header attached to a plurality of block units, information for determining a prediction procedure in the case where a mode without decoding of the motion vector is selected as a block mode.

  Furthermore, an apparatus to which the above method is applied is also disclosed.

  According to the above configuration of the present application, the determination as to whether or not the direct mode can be applied becomes clear. Also, the direct mode and its substitute mode can be effectively used, and the prediction efficiency can be improved and the amount of data can be reduced.

It is the figure which showed the example of the picture header data syntax in this invention. It is the figure which showed the 2nd example of the picture header data syntax in this invention. It is a figure explaining macroblock division. It is a figure explaining the principle of motion compensation. It is a figure which shows the block configuration utilized when encoding the presence or absence of the significant DCT coefficient contained in a macroblock. It is a figure which shows the macroblock structure which shows the block unit which performs a DCT process and an encoding process. It is a figure which shows the structure of the brightness | luminance block which shows the block unit which performs motion compensation. It is a figure explaining the production | generation means of a prediction motion vector. It is a figure showing the motion vector generation method for bidirectional prediction in Direct mode. It is the figure which showed the motion vector calculation method using two front direction reference frames. It is a figure showing a comparative example of decoding order and display order. It is the figure which showed the example of the switching procedure of the prediction method in this invention. It is a figure showing the 2nd example of a change procedure of a prediction method in the present invention. It is the figure which showed the whole structure of the substitute mode in this invention. It is a figure showing prediction mode selection processing of substitution mode in the present invention. It is a figure showing reference frame selection processing of substitution mode in the present invention. It is the figure which showed the selection process of the motion vector of the substitute mode in this invention. It is a figure showing an example of data syntax of a prediction frame. It is a figure showing an example of composition of a universal coding table. It is the figure which showed the example of the code table of the macroblock type in P-picture, and the code table of 8x8 block division | segmentation types. It is the figure which showed the example of the code table of the macroblock type in B-picture, and the code table of 8x8 block division | segmentation types. It is the figure which showed the example of the block diagram of the encoding process in this invention. It is the figure which showed the example of the block diagram of the decoding process in this invention. It is the figure which showed the 3rd example of the picture header data syntax in this invention. It is a figure showing the 3rd example of a change procedure of a prediction method in the present invention. It is the figure which showed the example of the prediction parameter calculator in the encoding apparatus which enforces the encoding method of this invention. It is the figure which showed the example of the prediction parameter calculator in the decoding apparatus which enforces the decoding method of this invention. FIG. 2 illustrates an example of a software encoder implementing the encoding method of the present invention. Fig. 2 shows an example of a software decoder implementing the decoding method of the present invention. It is the figure which showed the example of the optical disk which recorded the encoding bit stream produced | generated by the encoding method of this invention. It is the figure which showed the example of the apparatus which uses the encoding / decoding method of this invention.

  Hereinafter, embodiments of the present invention will be described using the drawings.

  The flow of processing from the frame header to the macroblock data will be sequentially described below.

FIG. 1 shows an example of frame header information. Also, an example of decoding processing of picture header data in C language will be shown below.
picture_layer ()
{
picture_structure
frame_number
reference_picture_selection_layer ()
if (coding_type () == B-picture) {
direct_mv_scale_bwd_dir [index]
if (direct_mv_scale_bwd_dir [index]) {// Future direction
direct_mv_scale_bwd [index]
for (index = 0; index <number of forward reference; index ++) {
direct_mv_scale_fwd_dir [index]
if (direct_mv_scale_fwd_dir [index]) // past direction
direct_mv_scale_fwd [index]
}
}
}
}
In picture_structure 20, the scan structure (frame / field) of each picture is shown. Frame_number 21 indicates the identification number of the frame. There are two major ways to attach this frame_number. One is the case where time information is included. In this case, in I, P-picture, the frame interval with the immediately preceding I or P-picture, in B-picture, the frame interval with the immediately preceding I or P-picture in the past is frame_number (generally, Temporal reference; called TR)). The second is the case of simply showing the order of decoding.

In reference_picture_selection_layer (), frame_number (reference picture set) of multiple reference frames usable for motion compensation processing of the current frame and their identification numbers are indicated. For example, when there are five reference frames, frame_number to index 0 to index 4 is assigned to the current picture of frame number 10 as follows.
Index 0: 9
Index 1: 8
Index 2: 7
Index 3: 6
Index 4: 5
When the picture type is P-picture, it is a forward reference picture set, and when it is B-picture, it is a forward reference picture set and backward reference picture. frame_number of (set) is decoded. At this time, the number of reference frames in the forward direction and the backward direction can be set separately, which may be different. When the picture type is I-picture or P-picture, the picture layer ends with byte alignment information (information for combining data delimiters into byte units) following the reference picture set information. Subsequent picture header data occurs when the picture type is B-picture. In this embodiment, it is considered that the information is described in a layer including higher-order network / communication related information.

  direct_mv_scale_bwd_dir 28 is information indicating whether the backward reference frame designated for the direct mode is located in the future or in the past with respect to the current frame. The back reference frame designated for direct mode is usually the backward reference picture assigned to index0. If data 28 indicates that a backward reference frame (here, backward reference picture assigned to index 0) is located in the past with respect to the current frame, direct mode can not be used and data 28 is backward. The direct mode can be used if it indicates that the current frame is located in the future with respect to the reference frame. Therefore, with this data 28, it can be clearly determined whether the backward reference picture of index 0 can be used for the direct mode. Further, when the direct mode can not be implemented, it is necessary to apply a substitute mode to be described later, and in preparation for the memory arrangement for that purpose, it is possible to promote the efficiency of the decoding operation. Furthermore, when frame_number does not have time information, it is possible to efficiently transmit information indicating the relationship between reference picture and current picture. Some picture position information related to the direct mode may be used other than the direct mode, and some may not be used. Regarding the latter, it is possible to avoid encoding by direct_mv_scale_bwd_dir.

  Specifically, as shown in FIG. 1, when the direct_mv_scale_bwd_dir indicates that the direct mode can be used, that is, it is positioned in the future direction with respect to the current frame, the data 26, 27, 29 is encoded. These data are not encoded when indicating that they are unusable.

  direct_mv_scale_bwd29 is information indicating a frame interval between the current picture and the backward reference picture of index 0 (see FIG. 9). direct_mv_scale_fwd_dir26 is information indicating whether the forward reference frame is located in the future or past with respect to the current frame, and direct_mv_scale_fwd 27 is information indicating the picture interval between the current picture and the forward reference picture (see FIG. 9). The direct_mv_scale_fwd needs to be encoded by the number of forward reference pictures specified in the reference_picture_selection_layer (22). However, since the forward reference frame needs to be located in the past with respect to the current frame in order to use the direct mode, direct_mv_scale_fwd 27 is omitted for the index in which direct_mv_scale_fwd_dir 26 indicates the future direction. direct_mv_scale_divider is information indicating the picture interval between the backward reference picture and the forward reference picture of index 0 (see FIG. 9). Therefore, this information also needs to be encoded by the number of forward reference pictures, but since it can be calculated from direct_mv_scale_fwd and direct_reference_bwd, it can be omitted. Also in this information, for the index whose direct_mv_scale_fwd_dir 26 indicates the future direction, the direct_mv_scale_fwd 27 is omitted.

  Also in the case of B-picture, the picture layer ends with byte alignment information (information for combining data delimiters into byte units).

Since direct_mv_scale_fwd and direct_mv_scale_bwd can also be used as picture_distance shown in FIG. 10, FIG. 2 shows a data syntax in which FIG. 1 is extended to motion vector coding of FIG.
picture_layer ()
{
picture_structure
frame_number
reference_picture_selection_layer ()
if (coding_type () == B-picture) {
for (index = 0; index <number of forward reference; index ++) {
direct_mv_scale_fwd_dir [index]
direct_mv_scale_fwd [index]
}
for (index = 0; index <number of backward reference; index ++) {
direct_mv_scale_bwd_dir [index]
direct_mv_scale_bwd [index]
}
}
}
The case of B-picture will be described. In this case, data 26 to 29 are encoded / decoded for all reference frames available in the current frame, but these data are used for the motion vector encoding process shown in FIG. Is also used as In FIG. 2 as well as FIG. 1, although direct_mv_scale_bwd_dir [0] plays a role of indicating whether or not the direct mode can be used, in FIG. 2 the process of FIG. 10 can be used by a combination of data 26 and 28 Indicates whether or not.

  The motion vector coding of FIG. 10 is effective when two reference frames corresponding to two motion vectors have the same direction with respect to the current frame. Therefore, if the values of data 26 and 28 corresponding to the index numbers of the two reference pictures selected in the block are a combination of two reference frames located in different directions with respect to the current frame, FIG. Instead of the motion vector coding method, each motion vector is individually coded / decoded by the method of FIG. On the other hand, if the values of data 26 and 28 corresponding to the index numbers of the two reference pictures selected in the block are a combination of two reference frames located in the same direction with respect to the current frame, from the current frame For distant motion vectors, the method of FIG. 10 is applied.

  So far, the backward reference picture used for the direct mode has been described as index0. However, it is also conceivable to select a number other than the index 0 from the backward reference picture set as the backward reference picture in the direct mode. For example, as in the direct_reference_idx_bwd 24 of FIGS. 1 and 2, it is possible to change the backward reference picture in units of frames by indicating the index number of the backward reference picture used for the direct mode in the picture layer. Furthermore, by setting direct_reference_idx_bwd24 not an index number itself but a value obtained by adding 1 to the index number, it means that the value "0" has the meaning that no backward reference frame can be used for direct mode in the backward reference picture set. It becomes possible.

  Next, the structure of macroblock data will be described using the macroblock layer syntax of FIG. 18 and the macroblock type code tables of FIGS. As a method used for encoding, a code combining Universal VLC (UVLC) using only one type of variable length code table, fixed length coding and variable length coding (different code tables are prepared for each coding element) It is possible to use a coding method and arithmetic coding (Witten et al., "Arithmetic Coding for Data Compression", Comm. Of the ACM, 30 (6), 1987, pp. 520-541), etc. , UVLC and arithmetic coding will be described as an example. Table 81 in FIG. 11 shows the configuration of UVLC, and the value of Xn is '0' or '1'. Table 82 shows an example of an actual variable length code table. As a specific method of arithmetic coding, a method is considered in which the meaning of each code is replaced by binary data of several bits, and each bit is coded according to an occurrence probability model indicating the occurrence probability of each 0 and 1. This method is called CABAC (Context-based Adaptive Binary Arithmetic Coding).

  FIG. 18 shows the syntax structure of macroblock data in B-picture. The macroblock data structure of B-picture and P-picture will be described using this figure. The description of the I-picture is omitted because it is not included in the features of the present application.

  In mb_skip_run11, run-length coding is performed for the number of consecutive SKIP modes (coding the number of consecutive 0s, and if the previous macro block type is not SKIP mode, the number of consecutive SKIP modes is zero) Data, which occurs only when using UVLC as a method of entropy coding. The SKIP mode is a macroblock type in which a prediction block image is used as it is as a reproduced block image without encoding a prediction error signal.

  The predicted block picture is synthesized in the direct mode in the method of cutting out the macroblock picture corresponding to the predicted vector position from the forward reference picture of index 0 in P-picture. This SKIP mode is a mode that is often selected in low rate coding, and particularly in B-picture, the probability of selection is high. Thus, the prediction performance of the direct mode directly affects the low rate coding performance.

  In the encoding method using CABAC, mb_skip_run11 is not used, and the SKIP mode is also handled as mb_type 12 (see the column of code number 0 in Tables 91 and 93). In mb_type 12, one mode is selected and encoded for each macroblock from the macroblock modes shown in Table 91 (P-picture) or Table 93 (B-picture).

  In Table 91, M and N of IntraM × N shown in code numbers 6 and 7 indicate small block sizes when performing spatial prediction, and M × N indicate small block sizes when performing motion compensation (see FIG. 7 shows mode 1 to mode 4). However, the mode indicated by code number 5 is not used in the case of CABAC.

  In Table 93, M and N of IntraM × N shown in code numbers 23 and 24 are small block sizes when performing spatial prediction, and M × N are small block sizes when performing motion compensation (mode 1 in FIG. 7) To <direct mode 4>, and direct indicates direct mode (direct (CBP == 0) indicates SKIP mode when CABAC is applied). Block 1 and Block 2 in Table 93 identify two small blocks in mode 2 or mode 3 in FIG. 7, and the prediction direction of each small block is Forward, Backward, Bi-predictive It indicates which of (two-way reference frame prediction).

  Here, the description of the direct mode is added. Although the direct mode is included in the selection candidates of mb_skip_run11 and mb_type 12, there are cases where the direct mode can not be applied in the method of applying the multiple reference frame function and the function of the two reference frames. Therefore, in the present invention, as shown in FIG. 12, a procedure of switching the prediction method in accordance with the condition is used.

  First, in direct_mv_scale_bwd_dir (FIG. 1) or direct_mv_scale_bwd_dir [0] (FIG. 2) in the picture header, it is checked whether the direct mode can be used for the current picture (301). If it is determined that the use is not possible in the process 301, a predicted macro block is created in a substitute mode (details will be described later) that does not require the forward MV of the collocated block (304). If it is determined in the process 301 that it can be used, the prediction method is selected for each 8x8 block. The 8 × 8 block is used here because the minimum unit of reference frame and prediction direction selection is 8 × 8 block in the block division method of FIG. 7. Specifically, it is checked whether or not the prediction mode having forward MV is applied to the collocated block corresponding to the 8x8 block (302). Then, if it is determined that it is applied, a prediction block is created in the direct mode (303), and if it is determined that it is not applied, a prediction block is created in the substitute mode (304) ).

  In process 302, for the collocated 8x8 block, when the prediction mode is the backward mode when the prediction mode is the intra mode, the value of direct_mv_scale_fwd_dir [index] for the forward reference picture is located in the backward direction (future direction) with respect to the current picture If the forward reference picture is not included in the forward reference picture set of the current picture, it is determined that the direct mode can not be used.

  In addition, in the process 302 of FIG. 12, although availability determination of direct mode is implemented in 8x8 block units, it is also possible to implement this in macroblock units. However, in this case, the direct mode is used only when all the prediction blocks in the macro block and all four 8x8 blocks in the case of the block division method of FIG. 7 can be used in the direct mode. It is determined that it is possible.

  FIG. 13 shows a procedure of switching the prediction method in the case where data 24 is added to the structure of the picture header. A point different from FIG. 12 is that the process 301 is changed to a process 305. Specifically, the index number of direct_mv_scale_bwd_dir to be checked is a value set in the data 24 of FIG. .

  It returns to the explanation of FIG. When 8x8 (split) is selected in mb_type 12, 8x8 Partition 13 is generated for each of four 8x8 small blocks 54-0 to 54-3 shown in mode 4 of FIG. Specifically, in the 8x8 Partition 18, one mode is selected and encoded for each 8x8 block from the 8x8 partition mode shown in Table 92 (P-picture) or Table 94 (B-picture).

  In Table 92, Intra indicated by code number 4 indicates spatial prediction, and M × N indicates the small block size (8 × 8 partition 1 to 8 × 8 partition 4 in FIG. 7) when performing motion compensation.

  In Table 94, Intra indicated by code number 13 indicates application of spatial prediction, M × N indicates small block size (8 × 8 partition 1 to 8 × 8 partition 4 in FIG. 7) when performing motion compensation, and direct indicates direct mode Is shown. Prediction in Table 94 indicates whether the prediction direction of each small block belonging to mode 4 in FIG. 7 is Forward (forward prediction), Backward (bidirectional prediction), or Bi-predictive (two reference frame prediction). .

  Even when the direct mode is selected in 8x8 Partition 18, the prediction method switching procedure similar to FIG. 12 or FIG. 13 can be applied. However, the prediction performance of the direct mode in the 8x8 partition is not as important as the direct mode MB. Therefore, it is also possible to apply a simpler method. For example, if it is determined in process 302 that the collocated block does not have Forward MV, then instead of process 304, Forward MV is set to 0 vector, forward reference picture and index of backward reference picture are set to 0. A method of generating a prediction block in the direct mode can be considered. At this time, when there is no backward reference picture, it is sufficient to generate a prediction block only by forward prediction. More simply, in the case where it is determined in process 302 that the collocated block does not have Forward MV, a method may be considered in which the direct mode is not selected on the encoding side.

  As for the encoding method of mb_type 12 and 8x8 Partition 13, when using UVLC, the code corresponding to the code number of Tables 91 to 94 is selected from Table 82 and encoded. When CABAC is used, the bit sequence shown in the Binarization column of Tables 91 to 94 is arithmetically encoded using a probability model of each bit.

  ref_index_fwd14 indicates the index number of the forward reference frame used for motion compensation, and is required for each divided block (51 to 54 in FIG. 7) in the macro block. The index number is selected from the forward reference picture set. However, if the forward reference picture set contains only one reference frame, this code does not occur if the block type or macroblock type is Skip, direct or intra, and if the block prediction is backward. .

  Also, when code number 5 in Table 91 is selected as mb_type in P-picture, this code does not occur because the forward reference picture of index 0 is automatically selected as the reference frame.

  A coding method will be considered, taking as an example the case where forward reference picture set has values of index 0 to index 4. In this example, index 0 to index 4 are assigned to code numbers 0 to 4, respectively. When using UVLC, codes corresponding to code numbers 0-4 are selected from Table 82 and encoded / decoded. When CABAC is used, binary data of 1 ', 01', 001 ', 0001', 00001 'is allocated to code number 0-4, and arithmetic coding is performed using a bit string and a probability model of each bit. Do.

  ref_index_bwd15 indicates the index number of the backward reference frame used for motion compensation, and is required for each divided block (51 to 54 in FIG. 7) in the macroblock.

  The index number is selected from the backward reference picture set. However, if the picture type is P-picture, if there is only one reference frame included in the backward reference picture set, if the block type or block type is skip, direct or intra, and the block prediction is forward. For the case of, this code does not occur. The encoding method is the same as ref_index_fwd, so the description will be omitted.

  mvd_fwd 16 occurs when mb_type 12 and 8x8 Partition 13 indicate that they are macroblocks with motion vectors accompanying forward (including bi-predictive), and are repeated by the number of forward MVs in the macroblock. Further, therefore, this data does not occur when mb_type 12 is IntraM × N, SKIP (P-picture) or direct (B-picture), or when 8 × 8 Partition 13 is intra or direct (B-picture). Further, this data does not occur even in the case of a divided block where block prediction is backward (B-picture). Similarly, mvd_bwd17 occurs when mb_type 12 and 8x8 Partition 13 indicate that they are macroblocks with motion vectors accompanying backward (including bi-predictive), and are repeated by the number of backward MVs in the macroblock. Therefore, when the picture type is P-picture, this data is not generated when mb_type 12 is IntraM × N, direct or when 8 × 8 Partition 13 is intra or direct. Also, this data does not occur in the case of a divided block whose block prediction is forward. CBP 18 is encoded data indicating whether or not quantized DCT coefficients (significant coefficients) other than '0' are included in 16 coefficients for the 24 DCT blocks shown in FIG.

  Residual () 19 indicates encoded data of significant quantized DCT coefficients. The coding process is omitted for blocks for which CBP indicates no significant coefficient. Therefore, when CBP is 0, Residual () does not occur. Furthermore, CBP 18 and Residual () 19 do not occur when mb_type 12 is direct (CBP == 0).

  Here, a method of generating the predicted motion vectors mvd_fwd 16 and mvd_bwd 17 shown above will be described with reference to FIG. 8 taking the division type of FIG. 7 as an example. Regarding the small blocks 54a-0, 54b-0, 54b-1, 54c-0, 54c-1 and 54d-0 to 54d-3 of the block 51-0, mode 4 (54) of mode 1 (51) of FIG. Uses the same prediction method. It is assumed that a small block to be encoded with a motion vector is 50. In these small blocks, for each of the horizontal and vertical components of the motion vector, an intermediate value is calculated with the motion vector of three blocks located at adjacent positions A, B and C as candidates, and the motion vector of the intermediate value is predicted motion vector I assume. However, it is conceivable that the block at position C may be located before encoding or outside the image depending on the encoding order and the macroblock position. In this case, the motion vector of the block located at the position D instead of the position C is used as one of the candidate motion vectors.

  When the blocks at positions A and D are located outside the image, prediction processing is performed using the motion vector as a '0' vector, and when positions D and B and C are located outside the image, , And performs prediction processing as a motion vector of a block at position A. At this time, when two of the three candidate blocks do not have a motion vector, one remaining candidate motion vector is set as a prediction motion vector. The two small blocks (52-0, 52-1) of mode 2 (52) and the two small blocks (53-0, 53-1) of mode 3 (53) are indicated by arrows shown in FIG. Let the motion vector of the block located at the root be the predicted value. Note that in motion vector coding in this method, only motion vectors of the same reference frame are used for prediction. Therefore, when the motion vector of the adjacent block is different from the reference frame selected in the coding block, it is treated as being located outside the image. The motion vector for the chrominance component is not encoded, and the motion vector of the luminance component is divided by 2 and used.

  The substitute mode (4x4 bi-predictive) which does not require Forward MV of a collocated block is demonstrated using FIGS. 14-17. Skip mode in B-picture using Direct mode as well as Direct mode is an important prediction scheme for high selectivity and high coding performance.

  However, in the method of giving freedom in selection of reference frames and coding procedures of each frame as in MPEG-4 Part 10, as shown in the description of FIGS. The frame or block which Direct mode does not function effectively occurs.

  This substitute mode has the effect of suppressing a drop in prediction performance and enhancing the prediction efficiency by switching and using it when the conventional Direct mode does not function effectively. Also, since the Direct mode uses the motion vector of the reference frame while this substitute mode uses the motion vector of the current frame, the motion vector memory is stored for encoding / decoding processing of subsequent frames. There is no need to store data in the memory, and the memory size can be reduced. Furthermore, in the substitute mode, since the motion vector scaling process shown in FIG. 9 is not necessary, the decoding process is simplified.

  The prediction procedure of this substitute mode is composed of four parts shown in FIG. First, the prediction direction is selected from bi-predictive, forward, backward in units of 8x8 blocks (610). The selection is performed using a block B83 immediately above the object 8x8 block C81 and a block immediately on the left B82. Next, selection of a reference frame necessary for implementation of the prediction mode selected in process 610 is performed (620). The selection is performed using a block B83 immediately above the object 8x8 block C81 and a block immediately on the left B82. Next, calculation processing of motion vectors corresponding to the selected prediction mode and reference frame is performed in 4 × 4 block units (630). Finally, the 4x4 prediction block is synthesized using the prediction mode selected in processing 610 and 620, the reference frame, and the motion vector calculated in processing 630. Then, the calculated motion vector and the index of the reference frame are stored for prediction motion vector prediction (640).

  As described above, by predicting each element data required for the prediction process from the information of surrounding blocks in the same frame, motion prediction according to the local feature becomes possible, and the prediction efficiency is improved. In addition, since only data of adjacent blocks in a frame is used, the amount of data to be stored for this substitute mode implementation is reduced. Details of the process are shown below.

  FIG. 15 shows the procedure of the prediction direction selection process of the process 610. First, it is checked whether any one of the 8x8 blocks immediately above and to the right of the target 8x8 block has Forward MV (611). Next, similarly, it is checked whether any one of the 8x8 blocks immediately above and to the right of the target 8x8 block has Backward MV (612). Then, if either the immediate upper or immediate left 8x8 block has forward MV or backward MV, or if neither the immediate or immediate left 8x8 block has forward MV or backward MV, then bi- Select the predictive (615). If the 8x8 block immediately above and to the left only has forward MV, it selects forward MV (616), and if it has only backward MV, backward MV (617) is selected.

  According to this procedure, bi-predictive prediction with the highest prediction efficiency is preferentially selected. In addition, when the information necessary for efficiently performing bi-predictive can not be obtained from the surrounding blocks, it is possible to select a prediction direction that is estimated to be optimal based on the information obtained from the surroundings. Furthermore, when sufficient information can not be obtained from the surroundings, controlling the direct mode to be highly effective for other prediction modes contributes to an improvement in prediction efficiency. Specifically, bi-predictive prediction of the zero motion vector with the forward reference and backward reference of index 0 (the frame most similar to the current frame) as a reference frame by combining with the processing in FIG. 16 and FIG. 17 described later. The procedure is to select the

  FIG. 16 shows the reference frame selection processing procedure of the processing 620. This process is performed separately for forward and backward. Although FIG. 16 shows the case of forward reference picture selection, the procedure is the same for backward reference picture.

  First, it is checked whether or not the 8x8 block immediately above the target 8x8 block and the 8x8 block directly to the left are all using the forward reference picture (621). If it is determined that any 8x8 block uses forward reference picture, the smaller index number is selected out of the forward reference pictures used in the two 8x8 blocks (623). If it is determined in process 621 that at least one of the 8x8 blocks does not use the forward reference picture, then either the 8x8 block directly above or to the left of the target 8x8 block is using the forward reference picture It is checked whether or not it is (622). If it is determined in process 622 that any 8x8 block uses the forward reference picture, the used forward reference picture is selected (625). If none of the 8 × 8 blocks use the forward reference picture in process 622, index 0 is selected (624).

  In this way, control is performed to select a smaller value among the Index numbers applied to the coding of the adjacent block. This is because, in the setting of reference frame candidates, a frame having a high correlation with the current frame is given a small index number. There are two methods of setting the index number: a method which is automatically set, and a method which is set at the time of encoding. In the former, smaller index numbers are given in order from the frame closer to the current frame. The latter is applied, for example, at a scene change time, and assigns a small index number to a frame of the same camera angle in the past that has been previously encoded. Thus, by selecting a small index number, the possibility of selecting an image close to the frame to be processed is increased.

  The motion vector calculation processing procedure of processing 630 will be described with reference to FIG. This process is separately performed for forward and backward in units of 4x4 blocks.

  First, it is checked whether any of the 4x4 block immediately above or right is located outside the image (631). If it is determined in the process 631 that any 4x4 block is located outside the image, the motion vector of the 4x4 block is set as a zero vector (625). If it is determined in process 631 that any 4x4 block is located in the image, then either the 4x4 block immediately above or immediately to the left can be used for the reference frame selected in process 620. It is checked whether or not it has any motion vector (632). If it is determined in process 632, that none of the 4x4 blocks have an available motion vector for the selected reference frame, then the motion vector of the 4x4 block is made zero vector (625). If it is determined at step 632 that any 4x4 block has an available motion vector to the selected reference frame, then any motion vector that the 4x4 block immediately above or immediately left has is processed. It is checked whether it is a zero vector to the reference frame selected at 620 (633). If it is determined in the process 633 that the motion vector of any 4x4 block is a zero vector for the reference frame, then the motion vector of the 4x4 block is set as a zero vector (625). If it is determined in the process 633 that none of the motion vectors of the 4 × 4 block is a zero vector for the reference frame, a motion vector is calculated by median prediction for the 4 × 4 block. Thus, the priority is given to selecting the zero vector because the Direct mode corresponds to being particularly effective in the background portion.

  The present invention also includes the following modifications.

  (1) In the present embodiment, although the use of the substitute mode is determined according to the situation of the collocated block as shown in FIG. 12, it is also conceivable to completely switch the direct mode to the substitute mode. In this method, in step 301, switching between the direct mode and the substitute mode is controlled in frame units or slice units (details will be described in the modification (4)). As a result, the number of selection candidates is increased, and the applicability to a scene subjected to a special effect is improved, so that the prediction efficiency is also improved. However, since there is a possibility that the calculation of the motion vector of the reference frame and the current frame FIG. 9 is performed by extrapolation processing, switching control of the two methods as shown in FIGS. The method to carry out is effective.

  (2) As for FIGS. 14 to 17, the detailed conditions are not limited as long as the overall process of generating the prediction direction, the reference frame, and the motion vector from the surrounding blocks is the same. For example, the present application also includes a method in which the description “either of 4 × 4 block immediately above or immediately left” is changed to “both 4 × 4 blocks immediately above and immediately left” in process 631. In addition, a method in which the number of blocks used at the time of mode selection is changed from two to three (used for creating a prediction vector) is included in the present application. The method of changing the number of blocks used in this mode selection from 2 to 3 is good in consistency with the motion vector estimation and leads to the improvement of the prediction efficiency, so it is effective under the condition that there is no strong restriction on the operation processing amount .

(3) In FIGS. 1, 2 and 12 and 13, if the collocated block has forward MV forward with respect to the current frame, the direct mode is applied regardless of the index number of forward reference picture for that forward MV. I have shown you how. However, the direct mode tends to be less effective when the forward reference picture for forward MV moves away from the current frame. Therefore, it is considered effective to apply the direct mode only when the index number of forward reference picture for forward MV is 0. The method will be described with reference to FIGS. 24 and 25. FIG. 24 shows the data syntax of the picture layer.
picture_layer ()
{
picture_structure
frame_number
reference_picture_selection_layer ()
if (coding_type () == B-picture) {
direct_reference_usable
if (direct_reference_usable) {
direct_mv_scale_bwd
direct_mv_scale_fwd
}
for (index = 0; index <number of forward reference; index ++) {
picture_distance_fwd_dir [index]
picture_distance_fwd [index]
}
for (index = 0; index <number of backward reference; index ++) {
picture_distance_bwd_dir [index]
picture_distance_bwd [index]
}
}
}
The case where the picture type is B-picture will be described. direct_reference_usable23 indicates whether the backward reference frame designated for direct mode is located in the future rather than the current frame, and whether the forward reference frame designated for direct mode is in the past before the current frame It is information. The back reference frame specified for direct mode is usually backward reference picture assigned to index 0, but with this information it can be clearly determined whether backward reference picture of index 0 can be used for direct mode .

  Also, the forward reference frame specified for direct mode is usually the forward reference picture assigned to index 0, but with this information it is clear whether forward reference picture for index 0 can be used for direct mode It can be judged. This data 23 is 0, that is, backward reference picture of index 0 is located forward (past direction) with respect to current picture, or forward reference picture of index 0 is backward (forward direction) with respect to current picture The picture interval information required for applying the direct mode does not need to be encoded / decoded because the direct mode can not be implemented for the located picture. Therefore, in this case, encoding / decoding of direct_mv_scale_fwd 2427 indicating the time interval of forward reference picture of current picture and index 0 and direct_mv_scale_bwd 2429 indicating the time interval of backward reference picture of current picture and index 0 is omitted. Data 26 to 29 are data used for bi-predictive motion vector coding shown in FIG. The method of use has been described in FIG. 2 and will not be described here.

  Note that direct_reference_usable 23 is information indicating only whether or not the backward reference frame designated for direct mode is located in the future relative to the current frame, and information (direct_mv_scale_fwd_dir) indicating the position of direct_mv_scale_fwd is encoded before data 2427. A method of conversion / decoding is also conceivable. When the forward reference picture is behind the current picture in FIG. 9, two motion vectors 122 and 121 are calculated by extrapolation.

  The handling of the direct mode will be described for the case of FIG. As shown in FIGS. 12 and 13, even when the direct mode is selected as the selection candidate for mb_skip_run11 and mb_type12 shown in FIG. 18, the direct mode is the mode that applies the multiple reference frame function and the function of two reference frames. Inapplicable cases may be considered. Therefore, in the present invention, a procedure of switching the prediction method according to the condition is used. The procedure is shown in FIG.

  First, in direct_reference_usable 23 in the picture header, it is checked whether the direct mode can be used for the current picture (306). In process 306, it is determined that the forward reference of the index0 is not available, that is, the forward reference of the index0 is located in the future in time with respect to the current picture or the backward reference of the index0 is located in the past in time with respect to the current picture. If so, a predicted macroblock is created in a substitute mode that does not require forward MV of the collocated block (304). If it is determined in the process 306 that it can be used, the prediction method is determined for each 8x8 block.

  The 8 × 8 block is used here because the minimum unit of reference frame and prediction direction selection is 8 × 8 block in the block division method of FIG. 7. Specifically, it is checked whether or not the prediction mode having Forward MV is applied to the collocated block corresponding to the 8x8 block (307). Then, if it is determined that it is applied, a prediction block is created in the direct mode (303), and if it is determined that it is not applied, a prediction block is created in the substitute mode (304) ). In process 307, for the collocated 8x8 block, direct mode if the prediction mode is intra mode, if the prediction direction is backward prediction, or if the forward reference picture is not the reference picture of index 0 included in the forward reference picture set of the current picture Determine that it can not be used.

  As in the case of FIG. 12, it is also possible to perform the usability determination of the direct mode in the process 307 in units of macroblocks. However, in this case, the direct mode is used only when all the prediction blocks in the macro block and all four 8x8 blocks in the case of the block division method of FIG. 7 can be used in the direct mode. It is determined that it is possible.

  Note that, as shown in the description of FIG. 24, it is also conceivable that only the condition that whether the forward reference of index 0 is located in the future temporally with respect to the current picture as direct_reference_usable 23 is indicated. In this case, there is a possibility that calculation of motion vectors by extrapolation calculation shown in the description of FIG. 24 is performed in the direct mode prediction of FIG.

  Furthermore, as shown in the above-mentioned modified example (1), it is conceivable that only the use determination condition of the direct mode is indicated by direct_reference_usable 23. Also in this case, if the use of the direct mode is specified and the forward reference is temporally located in the future, or if the backward reference is temporally located in the future, the direct mode of FIG. Motion vectors used for prediction are calculated by extrapolation.

  (4) In FIGS. 1, 2 and 24, although the data structure of the picture header is described with a limitation, the information is described in the header portion of the slice layer which is a group of plural macroblocks. In any case, the data structure of the present invention is applicable.

  In the method of data packetizing and transmitting compressed data in slice units, the data decoding procedure is determined by the information in the header portion of the slice layer. Therefore, the information of the present invention related to the decoding procedure is needed in the slice header part. Information indicating which macro block belongs to one slice is indicated in the communication packet header that controls upper communication / network related information, the header part of the file format, or the sequence header that determines the entire data configuration. And so on. The method of switching between the Direct mode and the substitute mode in slice units improves the freedom of selection and prediction efficiency as compared with the method of switching in frame units. However, in order to improve prediction efficiency, selection control in slice units is required, and therefore the amount of computation increases. Therefore, in an application requiring real-time processing, it can be said that switching control in frame units by frame structure is effective.

  The method of the present invention described so far can be applied to an image coding device / image decoding device using a dedicated circuit / chip and a software image coding device / software image decoding device using a general purpose processor.

  FIG. 28 shows an example of a portable terminal using an application processor as an example of the embedded software encoder / decoder. A host unit 2820 that mainly performs wireless communication processing, a camera input processing unit 2830 that processes an input signal from a camera, an application processor unit 2800 that performs application processing such as video encoding / decoding, and an output device that processes display data It is configured in 2840. At the time of encoding, an image captured by a camera is first converted to a YUV signal as shown in FIG. 3 by the camera input processing unit 2830 and input to the application processor unit 2800.

  The application processor unit 2800 encodes the input image into stream data as shown in FIG. 1 (or FIG. 2 or FIG. 24) and FIG. In the case of the embedded type, software (assembler code) for executing encoding processing (including the operations shown in the flowcharts of FIGS. 14 to 17) in the processing unit 2811 in the general-purpose processor 2810 is the internal RAM 2812 or the external RAM 2830. Are stored in advance. The area to be stored in advance with data used for prediction processing as shown in the flowcharts of FIGS. 14 to 17 (reference picture numbers of plural reference pictures and respective macroblocks, prediction direction, motion vector) is the internal RAM 2812 or It is secured in the external RAM 2830. The storage area arrangement of the assembler code and each data is designed with a balance of processor capacity, bus speed, estimated access frequency to assembler code and each data, and their capacity. Usually, the internal RAM has a faster access speed than the external RAM, and the external RAM has a larger mounted capacity than the internal RAM. Therefore, a data storage area or assembler code having a high access frequency and a small capacity is arranged in the internal RAM. At this time, the assembler code may be divided into an internal RAM and an external RAM.

  The encoded bit stream data is stored in the external RAM 2830 or a memory in the host unit 2820. It depends on the service for the portable terminal, such as the use of the encoded bit stream data, as to which is stored. At the time of decoding, encoded bit stream data is supplied from the host unit 2820 or the external RAM 2830 to the application processor unit 2800. The application processor unit 2800 decodes the input encoded bit stream data, converts the YUV reproduction image into an RGB image, and outputs the RGB image to the output device 2840. At this time, the YUV reproduction image may be temporarily stored in a frame memory in the external RAM or the internal RAM.

  Similarly to the case of the encoding process, also in the decoding process, software (assembler code) for performing the decoding process (including the operation shown in the flowcharts of FIGS. 14 to 17) in the processing unit 2811 in the general-purpose processor 2810 is , The internal RAM 2812 or the external RAM 2830 in advance. The area to be stored in advance with data used for prediction processing as shown in the flowcharts of FIGS. 14 to 17 (reference picture numbers of plural reference pictures and respective macroblocks, prediction direction, motion vector) is the internal RAM 2812 or It is secured in the external RAM 2830.

  FIG. 29 shows an example of a software encoder / decoder used for more general purpose applications.

  At the time of encoding, an input image is stored in a frame memory 2950, and the general-purpose processor 2900 reads information from here and performs encoding processing. The program for driving this general-purpose processor (including the operations shown in the flowcharts of FIGS. 14 to 17) is read from the storage device 2930 such as a hard disk or floppy (registered trademark) disk and stored in the program memory 2920 . The coding information output from the general-purpose processor is temporarily stored in the input / output buffer 2940 and then output as a coding bit stream.

  The processing memory 2910 stores data used for prediction processing as shown in the flowcharts of FIGS. 14 to 17 (a plurality of reference pictures and reference picture numbers of macroblocks, prediction directions, motion vectors). It is read by the general purpose processor according to the processing of the program. In addition, the general-purpose processor stores data in the processing memory according to the processing of the program.

  At the time of decoding, the input coded bit stream is temporarily stored in the input / output buffer 2940, and the general-purpose processor 2900 reads from here and decodes. The program for driving this general-purpose processor (including the operation shown in the flowcharts of FIGS. 14 to 17) is read from the storage device 2930 such as a hard disk or floppy (registered trademark) disk and stored in the program memory 2920 There is. The decoded reproduced image is temporarily stored in a frame memory 2950 and then output to a device that performs output processing. The processing memory 2910 stores data used for prediction processing as shown in the flowcharts of FIGS. 14 to 17 (a plurality of reference pictures and reference picture numbers of macroblocks, prediction directions, motion vectors). It is read by the general purpose processor according to the processing of the program. The general-purpose processor also stores data generated according to the processing of the program in a processing memory.

  The configuration of an image coding apparatus using a dedicated circuit and a dedicated chip is shown in FIG. The process flow of the coding process of one macroblock process will be described. First, the motion compensation process between the input macroblock image 201 and the decoded image (reference frame) of the encoded frame stored in the frame memory 210 is a combination of all macroblock types (8 × 8 Partition type) and candidate reference frames. , And is performed by the motion compensator 211, and the optimal macroblock type and 8x8 Partition type are selected. At this time, in the case of performing motion compensation in the Direct mode, it is necessary to acquire the prediction direction, the reference frame number, and the motion vector information from the MV predictor 215.

  FIG. 26 shows the internal structure of the MV predictor. Macroblock type (8x8 Partition type) indicating Direct mode, macroblock position information (block position information) and direct mode type (direct / alternative, controlled by motion compensator, alternative prediction is the substitution shown in FIGS. 14 to 17 When the mode is input, the switcher 2630 is activated through the switcher 2620. The switcher 2630 switches switches according to the type of direct mode.

  If the type of the direct mode is direct prediction, the motion vector calculator 2660 is activated. The motion vector calculation unit 2660 uses the information stored in the internal memory 2610 to calculate the prediction parameters shown in FIG. The calculated parameters are stored in the internal memory and notified to the motion compensator.

  If the type of the direct mode is alternative prediction, the alternative prediction unit 2640 is activated. The alternative prediction unit carries out the process shown in FIG. Specifically, the prediction mode selection unit 2641, the reference frame selection unit 2642, and the motion vector selection unit 2643 use the information stored in the internal memory 2610, respectively, as shown in FIGS. The prediction direction, the reference frame number, and the motion vector are calculated. These prediction parameters are stored in the internal memory and notified to the motion compensator.

  Return to the description of the motion compensator. After the optimum macroblock type is selected, the detected motion vector is notified to the MV predictor 215 together with the macroblock type, prediction direction information (forward / backward / bi-predictive), and reference frame number, and the contents of the internal memory 2610 are It will be updated (Macroblock type or 8x8 Partition type only if direct mode is selected).

  The motion vector prediction unit 2650 (activated by the switcher 2620) performs prediction processing shown in FIG. 8 for a block whose macroblock type and 8x8 Partition type are not direct, and calculates a differential motion vector. The calculated differential motion vector is output to the multiplexer 206 together with the macroblock type, 8x8 Partition type and reference frame number (if the direct mode is selected, the differential motion vector and reference frame number are not multiplexed ).

  Here, although it is assumed that the calculation of the differential motion vector is performed only for the optimal macroblock type (8x8 Partition type), the value of the differential motion vector and the code amount thereof are selected as the optimal macroblock type (8x8 Partition type) It is also conceivable to use it as an hourly evaluation value. In this case, for each combination of macroblock type (8x8 Partition type) and reference frame, a motion vector difference is calculated by the MV predictor.

  The predicted macroblock image 213 extracted from the reference frame generated by the motion compensation is input to the Intra / Inter determination processing unit 214. The Intra / Inter determination unit determines which of the intra mode and the inter mode is to be the final macroblock type, and notifies the determination information 218 to the multiplexer 206 and the MV predictor 215. When the determination information 218 is in the intra mode, the MV predictor 215 updates the stored data in the internal memory.

  The multiplexer uses the inter / inter mode judgment result, the inter mode macroblock type obtained from the MV predictor, the 8x8 Partition type, the reference frame number, and the differential motion vector (if the direct mode is selected, the differential The code shown in FIG. 18 is generated from the motion vector and the reference frame number) and multiplexed into a coded bit stream.

  When the macroblock type selected by the Intra / Inter determination unit is the inter mode, the predicted macroblock image is differentially processed by the difference unit 202 from the input macroblock image 201 of the current frame, and the difference macro A block image is generated. At this time, the predicted macroblock image is simultaneously output to the adder 209. When the macroblock type selected by the Intra / Inter determination unit is the intra mode, the predicted macroblock is not output to the difference unit 202 and the adder 209.

  The differential macroblock image or input macroblock image output from the difference unit 202 is first DCT-transformed. The block size of the DCT is generally 8 × 8 pixels in the conventional coding method, but recently, since the DCT conversion with the 4 × 4 pixel size is considered in MPEG-4 Part 10 (Advanced Video Coding) etc. The 4 × 4 DCT will be described as an example. The differential macroblock image is divided into 24 4 × 4 pixel blocks as shown in FIG. 6 and converted into 16 DCT coefficients by the DCT transformer 203, respectively. Each DCT coefficient is quantized by a quantizer 204 and encoded by a multiplexer 206.

  The multiplexer 206 multiplexes the macroblock data information as shown in FIG. 18 together with the header information as shown in FIG. 1, FIG. 2 or FIG. 24 to generate a coded bit stream. The quantized DCT coefficients are decoded into a differential macroblock image or a macroblock image in the inverse quantizer 207 and the inverse DCT unit 208 of the local decoder 220. When the prediction mode of the macro block is the inter mode, the difference macro block image is added to the predicted macro block image by the adder 209 and then synthesized in the frame memory 201. When the macroblock is in the intra mode, the restored macroblock image is synthesized in the frame memory 201.

  Although intra prediction is not performed in the intra mode of FIG. 22, the present invention is also applicable to a coding method for performing intra prediction. In this case, intra prediction may be performed in the Intra / Inter determination unit, but it is also possible to incorporate this process into the motion compensation unit. In particular, in a coding method in which a plurality of intra prediction types are prepared, such as MPEG-4 Part 10 (Advanced Video Coding), the intra prediction type can be handled in the same row as the inter prediction types, so the device configuration is simplified. In this case, the motion compensation unit 211 always supplies the difference prediction macroblock image 213 to the difference unit 202 and the adder 209. Further, since the determination information 218 is included in the macro block type information, the determination information 218 can be deleted, and the internal memory updating process in the MV prediction unit 215 accompanying the input of the determination information 218 is also omitted. In addition, intra prediction may be performed at the DCT coefficient level. This case can be coped with by incorporating a prediction process into the DCT transformation unit 203 and the IDCT transformation unit 208.

  The configuration of an image decoding apparatus using a dedicated circuit and a dedicated chip is shown in FIG. A process flow of the decoding process of one macroblock process will be described. First, the coding / decoding unit 501 analyzes the input coded data, and distributes motion vector related information and macroblock type information to the MV predictor 508 and quantized DCT coefficient information to the dequantizer 502.

  When the prediction mode of the macroblock is the inter mode, the block position information, the macroblock type, the 8x8 Partition type, the prediction direction information, the reference frame number, and the differential motion vector are input to the MV predictor 508 (macroblock type When the mode is the direct mode, only the macro block type and the macro block position information are input, and when the 8x8 partition type is direct, the reference frame number and the differential motion vector are not input for the 8x8 block) .

  FIG. 27 shows the internal structure of the MV predictor. When the macroblock type or 8x8 Partition type is direct, the type of direct mode (direct / alternative, controlled by motion compensator) in slice header information decoded by the decoding / decoding device 501 together with the macroblock position information or block position information Is input. When the macroblock position information (block position information) and the type of direct mode (direct / alternative, controlled by motion compensator) are input, the switcher 2630 is activated through the switcher 2620. The switcher 2630 switches switches according to the type of direct mode.

  If the type of the direct mode is direct prediction, the motion vector calculator 2660 is activated. The motion vector calculation unit 2660 uses the information stored in the internal memory 2710 to calculate prediction parameters shown in FIG. The calculated parameters are stored in the internal memory and notified to the motion compensator 504.

  If the type of the direct mode is alternative prediction, the alternative prediction unit 2640 is activated. The alternative prediction unit carries out the process shown in FIG. Specifically, the prediction mode selection unit 2641, the reference frame selection unit 2642, and the motion vector selection unit 2643 use the information stored in the internal memory 2710, respectively, as shown in FIGS. The prediction direction, the reference frame number, and the motion vector are calculated. These prediction parameters are stored in the internal memory 2710 and output to the motion compensator 504.

  When the macroblock type (8x8 Partition type) is not direct, together with the macroblock type (8x8 Partition type), the macroblock position information (block position information), the reference frame number and the differential motion vector are input, and the motion vector prediction is performed by the switcher 2620 Vessel 2750 is activated.

  The motion vector prediction unit 2750 performs the prediction process shown in FIG. 8 using the contents of the internal memory 2710 and the input data to restore the motion vector. The restored motion vector is output to the internal memory 2710 and the motion compensator 504 together with the prediction direction information and the reference frame number. The motion compensator 504 uses the input data and the reference picture in the frame memory 507 to generate a predicted macroblock image.

  Next, the inverse quantizer 502 and the inverse DCT 503 perform inverse quantization / inverse DCT processing on the encoded data relating to the prediction error signal for each 4 × 4 pixel block to reproduce a differential macroblock image. Then, the predicted macroblock image and the difference macroblock image are subjected to addition processing by the adder 505 to reproduce the macroblock image. The reproduced macroblock image is synthesized by the synthesizer 506 into the decoded frame image. Also, the decoded frame image is stored in the frame memory 507 for prediction of the next frame.

  When the macro block type is the intra mode, the inverse quantization unit 502 and the inverse DCT unit 503 apply inverse quantization / inverse DCT processing to the decoded quantized DCT coefficient information for each 4 × 4 pixel block, Play back a macroblock image. At this time, the contents of the internal memory 2710 of FIG. 27 are updated as the intra mode. Although intra prediction is not performed in this figure, the present invention can also be applied to a coding scheme in which a plurality of intra prediction types are prepared, such as MPEG-4 Part 10 (Advanced Video Coding). In this case, the motion compensation unit 504 includes the function of intra prediction, and the motion compensation unit always outputs a predicted macroblock image.

  Examples of storage media (recording media) on which the software encoders (FIGS. 14 to 17) shown in FIGS. 28 and 29 and the encoded bit stream generated by the encoders in FIGS. 22 and 26 are recorded are shown. Shown in.

  Digital information is recorded concentrically on a recording disk (magnetic disk or optical disk) 3000 capable of recording digital information. When a part 3001 of digital information recorded on the disc is taken out, slice header information 3010 including selection information (direct_reference_usable) 3011 of direct mode and substitute mode, SKIP mode information (mb_skip_run) 3021, 3031, 3041, 3051, Macroblock type information (mb_type, 8x8 partition) 3022, 3032, 3052, reference frame number and motion vector information (ref_index_few, ref_index_bwd, mvd_fwd, mvd_bwd) 3023, 3053, DCT coefficient and coding block pattern information (CBP, residual ()) 3024 and 3054 are recorded. Hereinafter, the data configuration will be described for the case where the frame type is B-picture and the direct mode is the substitute mode in the slice header. 3021 to 3024 and 3051 to 3054 indicate encoded data of macroblocks whose macroblock type is not Direct. This data configuration also applies when 8x8 Partition includes direct. However, in this case, the reference frame number and motion vector information for the 8x8 block for which the 8x8 Partition type is direct are not encoded, so these pieces of information are not included in 3023 and 3053. The prediction direction, the reference frame number, and the motion vector are calculated in process 2640 of FIG.

  The combination of 3031, 3032, and 3035 indicates the encoded data of the macroblock whose macroblock type is direct. In this case, reference frame numbers and motion vector information are not encoded.

  At the time of decoding, the prediction direction, the reference frame number, and the motion vector are calculated by the software decoder in FIG. 14 to FIG. 17 and in the dedicated decoding device in process 2640 in FIG. 3041 is an example of a skip macroblock, a macroblock type is direct, and DCT coefficient information does not exist.

  At the time of decoding, the prediction direction, the reference frame number and the motion vector are calculated by the software decoder in FIG. 14 to FIG. 17 in FIG. 14 and FIG. Becomes the reproduction macro block image as it is. As described above, by efficiently embedding a code indicating the direct mode as the macro block type in the storage medium, it is possible to synthesize a reproduction macro block image with a small amount of information.

  FIG. 31 shows a specific example of an apparatus for realizing the encoding method / decoding method of the present invention. Reads and decodes encoded bit stream recorded on optical disc 3101 (DVD-ROM, DVD-R, BD-ROM: Blu-ray (registered trademark) Disc ROM, CD-ROM / CD-R, etc.) which is a recording medium It is possible to implement the decoding method of the present invention also in the playback apparatus 3102 that In this case, the reproduced video signal is displayed on the television monitor 3103.

  A recording / reproducing apparatus 3112 that encodes terrestrial digital broadcasting or satellite digital broadcasting received from an antenna 3111 and records an encoded bit stream on an optical disc 3113 (DVD-RAM, DVD-RW, BD-RAM, CD-RW, etc.) It is also possible to implement the coding method of the present invention. The decoding method of the present invention can also be implemented in the recording / reproducing apparatus 3112 for decoding the coded bit stream recorded on the optical disc 3113. In this case, the reproduced video signal is displayed on the television monitor 3114.

  By incorporating software for the image encoding method / decoding method of the present invention into the personal computer 3121, it can be used as an image encoding / decoding device. This software is recorded on a storage medium (optical disk, floppy (registered trademark) disk, hard disk, etc.) 3122 which is a computer readable recording medium, and read by a personal computer for use. Furthermore, by connecting this personal computer to any communication line, it can be used as a video communication terminal.

  The decoding method in the present invention can also be implemented in the decoding device in the set top box 3132 connected to the cable 3131 for cable television or the satellite digital broadcast or terrestrial digital broadcast antenna, and the digital broadcast television monitor 3133 It is also conceivable to play on the Instead of the set top box, a decoding apparatus including the decoding method of the present invention may be incorporated in a television monitor.

  An apparatus or software encoder / decoder including the encoding method / decoding method of the present invention can also be implemented in the digital portable terminal 3141. As the implementation format, in addition to the transmission / reception type terminal having both the coding method and the decoding method, three implementation formats of a transmission terminal only for encoding and a reception terminal only for decoding can be considered.

  It is also possible to incorporate the encoding device / decoding device of the present invention into a camera 3151 for moving image shooting. In this case, the photographing camera has an encoding device and a recording device for recording the output from the encoding device on a recording medium, and records the encoded bit stream output from the encoding device on the recording medium. The recording medium may be an optical disc. If a camera is attached to the portable terminal, it is possible to encode the picked up image and send it out through an antenna.

  It is also possible to incorporate the encoding device / decoding device of the present invention into a video conferencing system 3161 with camera input. The video input from the camera is encoded into an encoded bit stream by the encoding device and distributed to the network 3162. The coded bit stream received from the network is decoded by the decoding device and displayed on the monitor. In this case, the means for realizing the encoding method and the decoding method of the present invention may be a software encoder / decoder rather than an encoder / decoder.

  By incorporating the encoding method and the decoding method of the present invention in these devices, it is possible to effectively use the direct mode and its substitute mode, and the prediction performance is improved.

  The header information of the present invention makes it possible to clearly determine whether or not the direct mode can be used. Furthermore, when the frame number does not have time information, it is possible to efficiently send information indicating the relationship between the reference frame and the current frame. Further, by the substitution mode of the present invention and its switching procedure, it is possible to improve the prediction performance when the direct mode can not be applied.

Claims (1)

A coding method for generating a coded bit stream for a moving image , comprising:
A first step of generating a first flag information used to determine whether the motion vector information of the decoding target block from the encoding side to the decoding side is the prediction mode that is not transmitted,
Prior SL prediction mode motion vector information is not transmitted to the decoding target block from the encoding side to the decoding side, the blocks adjacent to the current block a method of calculating a motion vector used for the decoding process the decoding side state and the second with both to determine the flag information, and a second step of generating the second flag information,
The method of calculating the motion vector includes
Even if the block at the same position as the decoding target block does not have a motion vector in the frame temporally behind the frame to which the decoding target block belongs on the decoding side, a plurality of adjacent blocks adjacent to the decoding target block A first motion vector calculation method capable of selecting one motion vector from the motion vectors it has;
The block located at the same position as the decoding target block in the frame located temporally behind the frame to which the decoding target block belongs on the decoding side has no motion vector, and the block positioned above the decoding target block and the decoding A second motion vector calculation method capable of selecting a zero vector of bi-directional prediction even when all blocks located on the left side of the target block are located outside the screen;
A third motion vector for calculating a motion vector of the block to be decoded based on a motion vector of a block located at the same position as the block to be decoded in a frame temporally behind the frame to which the block to be decoded belongs on the decoding side A coding method characterized in that it includes a calculation method .

Images