JP3380980B2 - Image encoding method, image decoding method, and image decoding device - Google Patents

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to an image coding method ,
The present invention also relates to an image decoding method and an image decoding device . In particular, for example, moving image data is recorded on a recording medium such as a magneto-optical disk or a magnetic tape and is reproduced and displayed on a display or the like, or a video conference system, a video telephone system, broadcasting equipment, a multimedia database. Used when transmitting moving image data from the transmitting side to the receiving side via a transmission line and displaying the received moving image data, or when editing and recording, etc. And an image decoding method , an image decoding method, and an image decoding apparatus .

[0002]

2. Description of the Related Art In a system for transmitting moving image data to a remote place, such as a video conference system or a video telephone system, for example, the image data is line-correlated or framed in order to use the transmission line efficiently. The compression coding is performed by using the inter-correlation.

A typical example of a high-efficiency coding method for moving images is the Moving Picture Experts Group (MPEG) (moving image coding for storage) method. This is ISO-IEC
/ JTC1 / SC2 / WG11, and it was proposed as a standard proposal, and a hybrid system combining motion compensation predictive coding and DCT (Discrete Cosine Transform) coding is adopted.

In MPEG, several profiles and levels are defined in order to support various applications and functions. The most basic is the main profile main level (MP @ ML (Main Profile
at Main Level)).

FIG. 38 shows MP @ M in the MPEG system.
The structure of an example of the encoder of L is shown.

Image data to be encoded is input to the frame memory 31 and temporarily stored. Then, the motion vector detector 32 reads out the image data stored in the frame memory 31, for example, in units of macroblocks composed of 16 pixels × 16 pixels, and detects the motion vector.

Here, in the motion vector detector 32, the image data of each frame is converted into an I picture (intra-frame coding), a P picture (forward prediction coding), or a B picture.
It is processed as one of the pictures (bidirectional predictive coding). It should be noted that which of I, P, and B pictures to process the images of the sequentially input frames is predetermined, for example (for example, I, B, P, B, P ,. .. are treated as B and P).

That is, the motion vector detector 32 refers to a predetermined reference frame in the image data stored in the frame memory 31, and the reference frame and the current encoding target. The motion vector of the macroblock is detected by pattern matching (block matching) with a small block (macroblock) of 16 pixels × 16 lines of the existing frame.

Here, in MPEG, there are four image prediction modes: intra coding (intra-frame coding), forward predictive coding, backward predictive coding, and bidirectional predictive coding.
There are types, I pictures are intra coded, P pictures are coded with either intra coding or forward predictive coding, and B pictures are intra coded, forward predictive coded, backward predictive coded, or both. It is encoded by any of the method predictive encodings.

That is, the motion vector detector 32 sets the intra coding mode as the prediction mode for the I picture. In this case, the motion vector detector 32
The motion vector is not detected, and the prediction mode (intra prediction mode) is output to the VLC (variable length coding) unit 36 and the motion compensator 42.

The motion vector detector 32 also performs forward prediction for P pictures to detect the motion vector. Further, the motion vector detector 32 compares the prediction error generated by performing the forward prediction with, for example, the variance of the macroblock to be encoded (macroblock of P picture). As a result of the comparison, when the variance of the macroblock is smaller than the prediction error, the motion vector detector 32 sets the intra coding mode as the prediction mode and outputs it to the VLC unit 36 and the motion compensator 42. Further, if the prediction error caused by performing the forward prediction is smaller, the motion vector detector 32 sets the forward prediction coding mode as the prediction mode, and the detected motion vector, the VLC unit 36, and the motion compensator 42 are set.
Output to.

Further, the motion vector detector 32 performs forward prediction, backward prediction, and bidirectional prediction for the B picture, and detects each motion vector. Then, the motion vector detector 32 uses forward prediction, backward prediction,
And a minimum prediction error (hereinafter, appropriately referred to as a minimum prediction error) among the prediction errors for bidirectional prediction, and the minimum prediction error and, for example, the variance of the macroblock to be encoded (the macroblock of the B picture). Compare with. As a result of the comparison, when the variance of the macroblock is smaller than the minimum prediction error, the motion vector detector 32 sets the intra coding mode as the prediction mode, and the VLC unit 3
6 and the motion compensator 42. If the minimum prediction error is smaller, the motion vector detector 32 sets the prediction mode in which the minimum prediction error is obtained as the prediction mode, and the VLC unit 3 with the corresponding motion vector.
6 and the motion compensator 42.

The motion compensator 42 is a motion vector detector 3
When receiving both the prediction mode and the motion vector from 2,
According to the prediction mode and the motion vector, the coded and already locally-decoded image data stored in the frame memory 41 is read out, and the read-out image data is used as predicted image data as the calculator 33.
And 40.

The calculator 33 reads from the frame memory 31 the same macroblock as the image data read from the frame memory 31 by the motion vector detector 32, and the difference between the macroblock and the predicted image from the motion compensator 42. Is calculated. This difference value is supplied to the DCT device 34.

On the other hand, the motion compensator 42 does not output the predicted image when it receives only the prediction mode from the motion vector detector 32, that is, when the prediction mode is the intra coding mode. In this case, the calculator 33 (calculator 40
The same applies to the macro block read from the frame memory 31 without performing any processing.
Output to.

In the DCT unit 34, DCT processing is performed on the output data of the arithmetic unit 33, and the resulting D is obtained.
The CT coefficient is supplied to the quantizer 35. Quantizer 35
Then, the amount of data accumulated in the buffer 37 (the amount of data stored in the buffer 37) (buffer feedback)
The quantization step (quantization scale) is set in correspondence with the DCT from the DCT unit 34 at the quantization step.
The coefficients are quantized. The quantized DCT coefficient (hereinafter, appropriately referred to as a quantized coefficient) is supplied to the VLC unit 36 together with the set quantization step.

In the VLC unit 36, the quantized coefficient supplied from the quantizer 35 is converted into a variable length code such as Huffman code, and output to the buffer 37. further,
The VLC unit 36 uses the quantization step from the quantizer 35,
Prediction mode from motion vector detector 32 (mode indicating which of intra coding (intra-picture predictive coding), forward predictive coding, backward predictive coding, or bidirectional predictive coding) is set and motion The vector is also variable-length coded, and the resulting coded data is stored in the buffer 37.
Output to.

The buffer 37 smoothes the data amount by temporarily storing the coded data from the VLC unit 36, and outputs it as a coded bit stream to, for example, a transmission line or records it on a recording medium.

The buffer 37 also outputs the data storage amount to the quantizer 35, and the quantizer 35 sets the quantization step in accordance with the data storage amount from the buffer 37. That is, the quantizer 35 uses the buffer 3
When 7 is about to overflow, the quantization step is increased, thereby reducing the data amount of the quantization coefficient. The quantizer 35 reduces the quantization step when the buffer 37 is likely to underflow, thereby increasing the data amount of the quantization coefficient. In this way, overflow and underflow of the buffer 37 are prevented.

The quantizing coefficient and the quantizing step output by the quantizer 35 are not limited to those of the VLC unit 36, but also of the inverse quantizer 3.
It is designed to be supplied to 8 as well. Inverse quantizer 38
Then, the quantized coefficient from the quantizer 35 is inversely quantized in accordance with the quantization step from the quantizer 35, and converted into the DCT coefficient. The DCT coefficient is supplied to the IDCT device (inverse DCT device) 39. I
In the DCT unit 39, the DCT coefficient is subjected to inverse DCT processing, and the data obtained as a result of the processing is supplied to the arithmetic unit 40.

In addition to the output data of the IDCT unit 39, the arithmetic unit 40 is supplied with the same data as the predicted image supplied to the arithmetic unit 33 from the motion compensator 42 as described above. The arithmetic unit 40 locally decodes the original image data by adding the output data (prediction residual (difference data)) of the IDCT unit 39 and the predicted image data from the motion compensator 42, and this local decoding Output image data (locally decoded image data) is output (however, when the prediction mode is intra-coding, the output data of the IDCT unit 39 passes through the arithmetic unit 40 and is directly output as the locally decoded image. It is supplied to the frame memory 41 as data).
The decoded image data is the same as the decoded image data obtained on the receiving side.

The decoded image data (locally decoded image data) obtained by the arithmetic unit 40 is supplied to and stored in the frame memory 41, and thereafter, inter-encoding (forward predictive coding, backward predictive coding, quantity direction prediction). It is used as reference image data (reference frame) for an image to be encoded.

Next, FIG. 39 shows an example of the configuration of an MP @ ML decoder in MPEG for decoding the encoded data output from the encoder of FIG.

The encoded bit stream (encoded data) transmitted through the transmission path is received by a receiver (not shown) or the encoded bit stream (encoded data) recorded on a recording medium is illustrated. It is reproduced by a reproducing device, is supplied to the buffer 101, and is stored therein.

The IVLC unit (inverse VLC unit (variable length decoder)) 102 reads the encoded data stored in the buffer 101 and performs variable length decoding to move the encoded data in macroblock units. Separate into vector, prediction mode, quantization step, and quantized coefficient. Of these data, the motion vector and the prediction mode are supplied to the motion compensator 107, and the quantization step and the quantized coefficient of the macroblock are supplied to the inverse quantizer 103.

The inverse quantizer 103 uses the quantization coefficient of the macroblock supplied from the IVLC unit 102 as I
Inverse quantization is performed according to the quantization step supplied from the VLC unit 102, and the resulting DCT coefficient is ID
Output to the CT device 104. The IDCT unit 104 inverts the DCT coefficient of the macroblock from the inverse quantizer 103 to the inverse DC.
Then, it is supplied to the arithmetic unit 105.

The output data of the motion compensator 107 as well as the output data of the IDCT device 104 is supplied to the arithmetic unit 105. That is, the motion compensator 107 sets the already-decoded image data stored in the frame memory 106 to I as in the case of the motion compensator 42 of FIG.
According to the motion vector and the prediction mode from the VLC unit 102, the arithmetic unit 1 is used as predicted image data.
Supply to 05. The calculator 105 outputs the output data (prediction residual (difference value)) of the IDCT unit 104 and the motion compensator 10
The original image data is decoded by adding it to the predicted image data from 7. The decoded image data is supplied to and stored in the frame memory 106. If the output data of the IDCT device 104 is intra-coded, the output data passes through the arithmetic unit 105 and is directly supplied to the frame memory 106 and stored as decoded image data. It

The decoded image data stored in the frame memory 106 is used as reference image data for image data to be subsequently decoded. Furthermore, the decoded image data is
The output reproduction image is supplied and displayed on, for example, a display (not shown).

In MPEG1 and MPEG2, since B picture is not used as reference image data, it is not stored in the frame memory 41 (FIG. 38) or 106 (FIG. 39) in the encoder or the decoder, respectively.

[0030]

[Problems to be Solved by the Invention]
The encoders and decoders shown in (1) are compliant with the MPEG1 / 2 standard, but are currently VO (Video Ob) which is a sequence of objects such as objects forming an image.
ISO-IEC for the method of encoding in units of
In / JTC1 / SC29 / WG11, MPEG
(Moving Picture Experts Group) 4 is in the process of standardization.

Meanwhile, with regard to MPEG4, since standardization work has been advanced mainly for use in the field of communication, GOP (Group Of Picture) defined in MPEG1 / 2 is defined in MPEG4. Therefore, if MPEG4 is used as a storage medium, it is expected that efficient random access will be difficult.

The present invention has been made in view of such a situation, and makes it possible to perform efficient random access.

[0033]

According to the image coding method of the present invention, a plurality of VOPs are grouped, and the VOPs of each group are grouped.
First addition step of adding absolute time information indicating the absolute time at which the encoding of the above is started in units of groups, and second precision time information generation step of generating second precision time information indicating relative time within the group with second precision And I-VOP, P
-VOP or B-VOP, detailed time information generation step of generating detailed time information representing the time from the second precision time information immediately before each display time to each display time with a precision finer than the second precision; -VOP, P-VO
As the information indicating the display time of the P or B-VOP, the second precision time information and the detailed time information are stored in the corresponding I-VO.
P, P-VOP, or a second addition step of adding to each of the B-VOPs, and in the second precision time information generation step, from the absolute time information to the predetermined VOP as the second precision time information for the predetermined VOP. The time up to the display time of is expressed in seconds precision or a predetermined VOP
The time from the display time of the I-VOP or the P-VOP displayed immediately before the display time of the predetermined VOP to the display time of the predetermined VOP is generated with the second precision, and the absolute time information is added to the I-VO.
Added to P, P-VOP, or B-VOP respectively
When the second precision time information and the detailed time information are added
The I-VOP, P-VOP, or B-VOP
It is characterized in that each display time .

The image decoding method of the present invention uses absolute time information.
I-VOP, P-VOP, or B-VOP respectively
Second precision time information and detailed time information added to this
By adding I-VOP, P-VOP, or B
-Display time calculation step for obtaining display time of each VOP, and I-VOP, P-VOP, or B-VOP
And a decoding step of decoding according to the corresponding display time, wherein the time from the absolute time information to the display time of the predetermined VOP is represented with second accuracy as the second accuracy time information for the predetermined VOP. Alternatively, it is characterized in that the time from the display time of the I-VOP or P-VOP displayed immediately before the predetermined VOP to the display time of the predetermined VOP is expressed in seconds accuracy.

The image decoding apparatus according to the present invention uses absolute time information.
I-VOP, P-VOP, or B-VOP respectively
Second precision time information and detailed time information added to this
By adding I-VOP, P-VOP, or B
-Display time calculating means for obtaining the display time of each VOP
And I-VOP, P-VOP, or B-VOP
And a decoding means for decoding according to the corresponding display time.
Yes, as the second precision time information for a given VOP,
Time from the time information to the display time of the specified VOP
In seconds precision, or immediately before a given VOP
Is it the display time of the displayed I-VOP or P-VOP?
, The time until the display time of the specified VOP is displayed with second accuracy.
It is characterized by the fact that it is used.

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

In the image coding method of the present invention, a plurality of VOPs are grouped, and absolute time information indicating the absolute time when the coding of the VOPs of each group is started is added in group units. Further, second precision time information representing relative time in the group with second precision is generated, and from the second precision time information immediately before the display time of each I-VOP, P-VOP, or B-VOP, the respective precision time information is displayed. Detailed time information is generated that represents the time until the display time with a precision finer than the second precision. And I-VOP, P-V
As information indicating the display time of OP or B-VOP,
Second precision time information and detailed time information correspond to IV
It is added to OP, P-VOP, or B-VOP, respectively. In this case, as the second precision time information about the predetermined VOP, the time from the absolute time information to the display time of the predetermined VOP is represented by the second precision or the predetermined VOP.
I-VOP or P-VOP displayed immediately before OP
The time from the display time of to the display time of the predetermined VOP is represented with second precision, and the absolute time information is
For each I-VOP, P-VOP, or B-VOP
Adds the second precision time information and detailed time information
The calculated time is I-VOP, P-VOP, or B-V
The display time of each OP is set.

In the image decoding method and the image decoding apparatus of the present invention, the absolute time information includes I-VOP and P-VO.
Second accuracy added to each P or B-VOP
By adding the time information and the detailed time information, the IV
When displaying OP, P-VOP, or B-VOP respectively
Is required, I-VOP, P-VOP, or B-V
The OP is decoded according to the corresponding display time. In this case, as the second precision time information for the predetermined VOP, the time from the absolute time information to the display time of the predetermined VOP is expressed in second precision, or the I-displayed immediately before the predetermined VOP. The time from the display time of the VOP or the P-VOP to the display time of the predetermined VOP, which is expressed in seconds, is used.

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

DETAILED DESCRIPTION OF THE INVENTION

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

FIG. 1 shows a configuration example of an embodiment of an encoder to which the present invention is applied.

The image (moving image) data to be encoded is V
O (Video Object) component 1 is input to VO component 1
Then, a VO, which is the sequence, is constructed for each object constituting the image input thereto, and is output to the VOP constructing sections 2 _{1 to} 2 _N. That is, when N VO # 1 to VO # N are configured in the VO configuration unit 1, the N VO # 1 to VO # N are output to the VOP configuration units 2 _{1 to} 2 _N , respectively.

Specifically, for example, when the image data to be encoded is composed of the sequence of the background F1 and the sequence of the foreground F2 which are independent, the VO constructing unit 1 changes the sequence of the foreground F2 to VO, for example. VOP as # 1
The sequence of the background F1 is output to the VOP constructing unit 2 ₂ as VO # 2 while being output to the constructing unit 2 ₁ .

If the image data to be encoded is, for example, a background F1 and a foreground F2 that have already been combined, the VO construction unit 1 divides the image into regions by a predetermined algorithm. Background F1 and foreground F
2 and take out VO as each sequence
To the corresponding VOP component 2 _n (where n = 1, 2, ...
.., N)

The VOP constructing unit 2 _n constructs a VOP (VO Plane) from the output of the VO constructing unit 1. That is, for example,
An object is extracted from each frame, and a minimum rectangle (hereinafter, appropriately referred to as a minimum rectangle) surrounding the object is set as a VOP. At this time, the VOP component 2 _n
The VOP is configured such that the number of horizontal and vertical pixels is, for example, a multiple of 16. When the VO constructing unit 2 _n constructs the VOP, the VO constructing unit 2 _n outputs the VOP to the VOP encoding unit 3 _n .

Further, the VOP constructing section 2 _n includes size data (VOP size) representing the size (for example, the horizontal and vertical lengths) of the VOP and the position of the VOP in the frame (for example, the upper leftmost part of the frame). The offset data (VOP offset) representing the coordinate when the origin is the origin is detected, and these data are also supplied to the VOP encoding unit 3 _n .

The VOP encoding unit 3 _n outputs the output of the VOP forming unit 2 _n to, for example, MPEG or H.264. It is encoded by a method compliant with standards such as H.263 and the resulting bit stream is output to the multiplexing unit 4. The multiplexing unit 4 uses the VO
The bit streams from the P coding units 3 _{1 to} 3 _N are multiplexed, and the resulting multiplexed data is transmitted, for example, via a terrestrial wave, a satellite line, a CATV network, or another transmission line 5, or For example, it is recorded on a recording medium 6 such as a magnetic disk, a magneto-optical disk, an optical disk, a magnetic tape or the like.

Here, VO and VOP will be described.

VO is a sequence of objects (objects) forming a composite image when a sequence of the composite image exists, and VOP means VO at a certain time. That is, for example, when there is a composite image F3 that is composed by combining the images F1 and F2, the images in which the images F1 and F2 are arranged in time series are VO, respectively, and the images F1 and F2 at a certain time.
Are VOPs. Therefore, a VO can be said to be a set of VOPs of the same object at different times.

For example, when the image F1 is used as the background and the image F2 is used as the foreground, the composite image F3 uses the key signal for extracting the image F2 and the images F1 and F1.
Although it is obtained by synthesizing the two, the VOP of the image F2 in this case includes not only the image data (luminance signal and color difference signal) forming the image F2 but also its key signal as appropriate.

The sequence of image frames does not change in either size or position, but VO
The size and position may change. That is, the same VO
The size and position of the VOP that composes may differ depending on the time.

Specifically, FIG. 2 shows the background image F1.
And a foreground image F2.

The image F1 is, for example, a photograph of a certain natural landscape, and the sequence of the entire image is one VO (referred to as VO # 0). Also, the image F2
Is, for example, a photograph of a person walking, and the smallest rectangular sequence surrounding the person is one V
It is O (denoted as VO # 1).

In this case, since VO # 0 is a landscape image, basically, both its position and size do not change, like a frame of a normal image. On the contrary,
Since VO # 1 is an image of a person, its size and position change as the person moves to the left or right, or moves to the front side or the back side in the drawing. Therefore, FIG.
Represents VO # 0 and VO # 1 at the same time, but the position and size of VO may change over time.

Therefore, the VOP coding unit 3 _n in FIG. 1 outputs, in the bit stream output from the VOP coding unit 3 _n , information on the position (coordinates) and the size of the VOP in a predetermined absolute coordinate system, in addition to the VOP-coded data. It is designed to be included. In FIG. 2, the vector indicating the position of the VOP (image F1) at a certain time, which constitutes VO # 0, is O.
ST0 and V of VO # 1 at the same time as ST0
A vector representing the position of OP (image F2) is OST1 and
Each is represented.

Next, FIG. 3 shows an example of the structure of the VOP coding unit 3 _n shown in FIG. 1, which realizes scalability.
That is, MPEG has introduced a scalable coding method that realizes scalability corresponding to different image sizes and frame rates, and the VOP coding unit 3 _n shown in FIG. 3 can realize such scalability. It is made possible.

The VOP (image data) from the VOP constructing unit 2 _n , its size data (VOPsize), and offset data (VOP offset) are all supplied to the image hierarchizing unit 21.

The image hierarchization unit 21 generates image data of one or more hierarchies from VOPs (performs one or more hierarchies of VOPs). That is, for example, in the case of performing spatial scalability encoding, the image layering unit 21 outputs the image data input thereto as it is as image data of an upper layer (upper layer), and at the same time, outputs the image data. Is reduced (the resolution is lowered) by thinning out the number of pixels forming the image, and this is output as image data of a lower layer (lower layer).

It is also possible to use the input VOP as lower layer data, increase the resolution of the VOP by some method (increase the number of pixels), and use this as upper layer data. Is.

The number of layers can be 1, but in this case scalability is not realized. In this case, the VOP coding unit 3 _n is composed of only the lower layer coding unit 25, for example.

Further, the number of layers can be three or more, but here, for simplicity, a case of two layers will be described.

For example, when performing coding of temporal scalability (temporal scalability), the image layering unit 21 alternately outputs image data as lower layer data or upper layer data, for example, according to time. That is, for example, in the image hierarchization unit 21, the VOPs that form a certain VO are VOP0, VOP1, VOP.
If input in the order of 2, VOP 3, ..., V
OP0, VOP2, VOP4, VOP6, ... Are used as lower layer data, and VOP1, VOP3,
VOP5, VOP7, ... Are output as upper layer data. In the case of temporal scalability, the VOPs decimated in this way are only data of the lower layer and the upper layer, and the image data is not enlarged or reduced (conversion of resolution) (however, not shown). , It is also possible to do).

Further, the image hierarchizing unit 21 uses, for example, SN
When encoding R (Signal to Noise Ratio) scalability, input image data is output as it is as upper layer data or lower layer data, respectively. That is, in this case, the image data of the lower layer and the image data of the upper layer are the same data.

Here, with respect to the spatial scalability in the case of performing coding for each VOP, the following three types can be considered, for example.

That is, for example, as a VOP, as shown in FIG.
Assuming that a composite image composed of the images F1 and F2 as shown in FIG. 4 is input, the first spatial scalability is as shown in FIG. 4, and the entire input VOP (FIG. 4A)
Is an upper layer (Enhancement Layer), and the entire VOP is reduced (FIG. 4B) is a lower layer (Base Layer).

The second spatial scalability is as shown in FIG. 5, in which a part of the objects constituting the input VOP (FIG. 5 (A) (here, the part corresponding to the image F2)). Pull out (note that this kind of pull out is
For example, it is performed in the same manner as in the case of the VOP configuration unit 2 _n , and thus the object extracted by this can also be considered as one VOP), which is an upper layer and the entire VOP is reduced. (FIG. 5 (B)) is the lower layer.

Further, the third spatial scalability is
As shown in FIG. 6 and FIG. 7, an object (VOP) forming the input VOP is extracted, and an upper layer and a lower layer are generated for each object. Note that FIG. 6 shows the background (image F1) that constitutes the VOP of FIG.
7 shows the case where the upper layer and the lower layer are generated, and FIG. 7 shows the case where the upper layer and the lower layer are generated from the foreground (image F2) forming the VOP of FIG.

Which of the above scalability is used is determined in advance, and the image hierarchy unit 2
1 performs VOP layering so that encoding can be performed according to the predetermined scalability.

Further, the image hierarchizing unit 21 generates the lower layer and the upper layer from the VOP size data and the offset data (respectively referred to as initial size data and initial offset data, respectively) input thereto. Offset data representing the position of the VOP in a predetermined absolute coordinate system and size data representing the size thereof are calculated (determined).

Here, the V of the lower layer and the V of the upper layer are
A method of determining OP offset data (position information) and size data will be described, for example, in the case where the above-described second scalability (FIG. 5) is performed.

In this case, the offset data FPOS_B of the lower layer is, for example, as shown in FIG. 8A, the image data of the lower layer is enlarged (upsampling) based on the difference in the resolution and the resolution of the upper layer.
When it is done, that is, the enlargement ratio that matches the image of the lower layer with the size of the image of the upper layer (the reciprocal of the reduction ratio when the image of the lower layer is generated by reducing the image of the upper layer) When the image is enlarged by (hereinafter, appropriately referred to as a magnification FR), the offset data in the absolute coordinate system of the enlarged image is determined so as to match the initial offset data. In addition, the size data FSZ_ of the lower layer
Similarly, B is determined so that the size data of the enlarged image obtained when the image of the lower layer is enlarged by the magnification FR matches the initial size data. That is, the offset data FPOS_B or the size data FSZ_B is determined to be FR times the initial value, or to match the initial offset data or the initial size data.

On the other hand, the offset data FP of the upper layer
The OS_E is, for example, as shown in FIG. 8B, a minimum rectangle (VO) surrounding an object extracted from the input VOP.
For example, the coordinate of the upper left vertex of P) is obtained based on the initial offset data and is determined to this value. Further, the size data FPOS_E of the upper layer is determined to have, for example, the horizontal and vertical lengths of the smallest rectangle surrounding the object extracted from the input VOP.

Therefore, in this case, the offset data FPOS_B and the size data FPOS_B of the lower layer
Is converted according to the scaling factor FR (offset data FPOS_B or size data FPOS_B after conversion).
Are respectively referred to as conversion offset data FPOS_B or conversion size data FPOS_B), and in an absolute coordinate system, at a position corresponding to the conversion offset data FPOS_B, an image frame having a size corresponding to the conversion size data FSZ_B is considered, F the layer image data
A magnified image of only R times is placed (see FIG. 8).
(A)), offset data FPOS_E and size data FPOS of the upper layer in the absolute coordinate system
When the images of the upper layer are similarly arranged according to _E (FIG. 8B), the pixels forming the enlarged image and the pixels forming the image of the upper layer correspond to each other at the same position. Will be placed in. That is, in this case, for example, in FIG. 8, the part of the person who is the upper layer image and the part of the person in the enlarged image are arranged at the same position.

In the first and third scalability, similarly, the corresponding pixels forming the enlarged image of the lower layer and the image of the upper layer are arranged at the same position in the absolute coordinate system. Then, the offset data FPOS_B and FPOS_E and the size data FSZ_B and FSZ_E are determined.

Returning to FIG. 3, the upper layer image data and the offset data F generated by the image layering unit 21.
The POS_E and the size data FSZ_E are delayed by the delay circuit 22 by the processing time in the lower layer encoding unit 25, which will be described later, and are supplied to the upper layer encoding unit 23. Further, the lower layer image data, the offset data FPOS_B, and the size data FSZ_B are supplied to the lower layer encoding unit 25. Also, the magnification FR
Is supplied to the upper layer encoding unit 23 and the resolution converting unit 24 via the delay circuit 22.

In the lower layer encoding unit 25, the image data of the lower layer is encoded, and the encoded data (bit stream) obtained as a result is offset data FPO.
The S_B and size data FSZ_B are included and supplied to the multiplexing unit 26.

Further, the lower layer encoding unit 25 locally decodes the encoded data, and as a result, outputs the lower layer image data which is the local decoding result to the resolution converting unit 24.
The resolution conversion unit 24 restores the image data of the lower layer from the lower layer encoding unit 25 to the original size by enlarging (or reducing) the image data according to the magnification FR, and the enlarged image obtained by this is converted into an upper image. Layer coding unit 23
Output to.

On the other hand, in the upper layer encoding unit 23, the image data of the upper layer is encoded, the encoded data (bit stream) obtained as a result includes the offset data FPOS_E and the size data FSZ_E, and the multiplexing unit 26. In the upper layer encoding unit 23, the encoding of the upper layer image data is performed using the enlarged image supplied from the resolution conversion unit 24 as the reference image.

In the multiplexing unit 26, the upper layer coding unit 2
3 and the outputs of the lower layer encoding unit 25 are multiplexed and output.

The lower layer coding unit 25 supplies the lower layer size data FSZ_B, the offset data FPOS_B, the motion vector MV, the flag COD, etc. to the upper layer coding unit 23. The conversion unit 23 is configured to perform processing while referring to these data as necessary. The details will be described later.

Next, FIG. 9 shows a detailed configuration example of the lower layer encoding unit 25 of FIG. In the figure, the same reference numerals are given to the portions corresponding to the case in FIG. That is, the lower layer encoding unit 25 is basically configured similarly to the encoder of FIG.

The image data from the image hierarchizing unit 21 (FIG. 3), that is, the VOP of the lower layer is supplied to and stored in the frame memory 31 as in the case of FIG.
The motion vector detector 32 detects a motion vector in macroblock units.

However, the motion vector detector 32 of the lower layer encoding unit 25 is supplied with the size data FSZ_B and the offset data FPOS_B of the VOP of the lower layer, in which the size data FSZ_B and The motion vector of the macro block is detected based on the offset data FPOS_B.

That is, as described above, since the size and the position of the VOP change depending on the time (frame),
In detecting the motion vector, it is necessary to set a coordinate system that serves as a reference for the detection and detect the motion in the coordinate system. Therefore, here, the motion vector detector 32 uses the above-mentioned absolute coordinate system as a reference coordinate system, and according to the size coordinate system FSZ_B and the offset data FPOS_B, the VOP and the reference image to be encoded are set in the absolute coordinate system. Place the VOP to
It is designed to detect a motion vector.

The detected motion vector (MV)
Is supplied to the VLC unit 36 and the motion compensator 42 together with the prediction mode, and is also supplied to the upper layer encoding unit 23 (FIG. 3).

Also, when performing motion compensation, the motion compensator 42 also needs to detect the motion in the reference coordinate system as described above, and therefore the size data FSZ_B and the offset data FPOS_.
B is supplied.

The detected VOP of the motion vector is shown in FIG.
In the same way as in the case of 8, the VLC unit 3
6 is supplied. Similarly to the case in FIG. 38, the VLC unit 36 is supplied with the quantization coefficient, the quantization step, the motion vector, and the prediction mode, and also the size data FSZ_B and the offset data FPOS_B from the image layering unit 21. Is provided, where all of this data is variable length encoded.

The VOP in which the motion vector is detected is encoded as described above, and also locally decoded as in the case of FIG. 38 and stored in the frame memory 41. This decoded image is used as a reference image as described above, and is also output to the resolution conversion unit 24 (FIG. 3).

Note that in MPEG4, MPEG1
Unlike 2 and 2, since the B picture (B-VOP) is also used as the reference image, the B picture is also locally decoded and stored in the frame memory 41 (however, at the present time, the B picture is It is only used as a reference image for upper layers).

On the other hand, the VLC device 36, as described with reference to FIG. 38, has I, P, B pictures (I-VOP, P-VO).
For macroblocks of (P, B-VOP), it is determined whether or not to be skip macroblocks, and flags COD and MODB indicating the determination result are set. The flags COD and MODB are also variable length coded and transmitted. Further, the flag COD indicates that the upper layer encoding unit 2
3 is also supplied.

Next, FIG. 10 shows an example of the configuration of the upper layer coding section 23 of FIG. Note that, in the figure, portions corresponding to those in FIG. 9 or FIG. 38 are denoted by the same reference numerals. That is, the upper layer encoding unit 23
The configuration is basically the same as that of the lower layer encoding unit 25 of FIG. 9 or the encoder of FIG. 38, except that a frame memory 52 is newly provided.

The image data from the image hierarchizing unit 21 (FIG. 3), that is, the VOP of the upper layer is supplied to and stored in the frame memory 31 as in the case of FIG.
The motion vector detector 32 detects a motion vector in macroblock units. In this case, too,
In the motion vector detector 32, as in the case of FIG. 9, in addition to the VOP of the upper layer, its size data FSZ
_E and offset data FPOS_E are supplied, and the motion vector detector 32, based on this size data FSZ_E and offset data FPOS_E, in the motion vector detector 32, detects the VOP of the upper layer in the absolute coordinate system. The arrangement position is recognized, and the motion vector of the macroblock is detected.

Here, the motion vector detector 32 in the upper layer encoding unit 23 and the lower layer encoding unit 25.
Then, as described with reference to FIG. 38, the VOP is processed in accordance with the predetermined sequence set in advance, and the sequence is set here, for example, as follows.

That is, in the case of spatial scalability, as shown in FIG. 11 (A) or FIG. 11 (B),
The VOP of the upper layer or the lower layer is, for example, P,
Processing is performed in the order of B, B, B, ... Or I, P, P, P ,.

In this case, the first V of the upper layer
The P picture (P-VOP), which is an OP, is encoded using, for example, a VOP of a lower layer at this time (here, an I picture (I-VOP)) as a reference image. Further, the B picture (B-VOP) that is the second and subsequent VOPs of the upper layer is encoded using, for example, the VOP of the immediately preceding upper layer and the VOP of the lower layer at the same time as the reference image. That is, here
The B picture of the upper layer is used as a reference image when other VOPs are coded like the P picture of the lower layer.

As for the lower layer, for example, M
PEG 1 or 2, or H.264. Encoding is performed as in the case of H.263.

Since it is considered that the SNR scalability is when the scaling factor FR in the spatial scalability is 1, the SNR scalability is processed in the same manner as the case of the spatial scalability described above.

In the case of temporal scalability, that is, for example, as described above, VO is VOP0, VO
It is composed of P1, VOP2, VOP3, ...
1, VOP3, VOP5, VOP7, ... Are upper layers (FIG. 12A), and VOP0, VOP2, VO
When P4, VOP6, ... Are set as lower layers (FIG. 12 (B)), as shown in FIG. 12, the VOP of the upper layer or the lower layer is, for example, B, B,
B, ... Or I, P, P, P ,.

In this case, the first V of the upper layer
OP1 (B picture) is, for example, a VOP of a lower layer.
It is coded using 0 (I picture) and VOP2 (P picture) as reference images. Also, the upper layer 2
The th VOP3 (B picture) is, for example, the VOP1 of the upper layer coded as a B picture immediately before it,
And VOP4 (P picture) of the lower layer, which is an image at the time (frame) next to VOP3, is used as a reference image for encoding. Third VOP in the upper layer
Similarly to VOP3, 5 (B picture) is, for example, the VO of the upper layer coded immediately before it as a B picture.
Coding is performed by using VOP6 (P picture) of the lower layer, which is an image at the time (frame) next to P3 and VOP5, as a reference image.

As described above, with respect to a VOP of a certain layer (here, the upper layer), another layer (scalable layer) (here, the lower layer) is used as a reference image for encoding P and B pictures. VOPs can be used. As described above, when the VOP of a certain layer is used as a reference image for encoding the VOP of another layer, that is, here, the VOP of the lower layer is referred to for predictively encoding the VOP of the upper layer. When used as an image, the motion vector detector 32 of the upper layer encoding unit 23 (FIG. 10) uses the flag ref_ indicating that fact.
The layer_id (when the number of layers is 3 or more, the flag ref_layer_id represents the layer to which the VOP used as the reference image belongs) is set and output.

Furthermore, the motion vector detector 32 of the upper layer encoding unit 23 uses the flag ref_ regarding VOP.
According to the layer_id, a flag ref_select_ indicating which layer of the VOP is used as the reference image for forward predictive coding or backward predictive coding, respectively.
The code (reference image information) is also set and output.

That is, for example, the upper layer (Enhancement
Layer) P picture is decoded immediately before (local decoding)
If a VOP belonging to the same layer as the reference image is encoded as a reference image, the flag ref_sel
The ect_code is set to “00”. In addition, the P picture is a V that is displayed immediately before and belongs to a different layer (here, lower layer) (Reference Layer).
When encoded using OP as a reference image, the flag ref_select_code is set to "01".
Furthermore, if the P picture is coded using the VOP displayed immediately after that and belonging to a different layer as the reference image, the flag ref_select_cod
e is set to "10". Also, when a P picture is coded using VOPs of different layers at the same time as reference pictures, the flag ref_select is used.
_Code is set to "11".

On the other hand, for example, a B picture of an upper layer uses a VOP of a different layer at the same time as that of a VOP belonging to the same layer as the reference image for forward prediction and decoded immediately before that. Is coded using as a reference image for backward prediction,
The flag ref_select_code is set to "00". In addition, a B picture of an upper layer uses a VOP belonging to the same layer as the reference image for forward prediction, and a VOP displayed immediately before that belonging to a different layer is used as a reference image for backward prediction. Flag is used as a flag ref_select
_Code is set to "01". Further, the B picture of the upper layer uses the VOP which is decoded immediately before and belongs to the same layer as the reference image for forward prediction, and the VOP which is displayed immediately after that and belongs to a different layer is backward. When coded using as a reference image for prediction, the flag ref_select_c
The ode is set to "10". In addition, a B picture of an upper layer, which is displayed immediately before that, belongs to a layer different from that, is used as a reference image for forward prediction,
In addition, when the VOP displayed immediately after that and belonging to a layer different from that is coded using as a reference image for backward prediction, the flag ref_select_code is used.
Is set to "11".

Here, the predictive coding method described in FIGS. 11 and 12 is one example, and the forward predictive coding,
Which VOP of which layer is used as a reference image in backward predictive coding or bidirectional predictive coding is
For example, it can be freely set within the range described above.

In the above case, for convenience,
Although the terms "spatial scalability", "temporal scalability" and "SNR scalability" are used, the flag ref_select_code allows
When setting a reference image used for predictive coding, spatial scalability, temporal scalability, SNR
It becomes difficult to clearly distinguish scalability.
That is, conversely, the flag ref_select_co
By using de, it becomes unnecessary to make the above distinction of scalability.

Here, if the above scalability is associated with the flag ref_select_code, for example, the following is obtained. That is, for P pictures, the flag ref_select_code is "1".
The case of “1” is the case where the VOP at the same time of the layer indicated by the flag ref_layer_id is used as the reference image (reference image for forward prediction), and therefore this corresponds to the spatial scalability or the SNR scalability. Then, the flag ref_select_
Corresponding to temporal scalability except when the code is "11".

For B pictures, the flag re
The case where f_select_code is "00" is also the case where the VOP at the same time of the layer indicated by the flag ref_layer_id is used as a reference image for forward prediction, and this corresponds to the spatial scalability or the SNR scalability. Then, except for the case where the flag ref_select_code is "00", it corresponds to temporal scalability.

Note that a different layer (here, lower layer) is used for VOP predictive coding of the upper layer.
When the VOP at the same time is used as the reference image, there is no motion between the two, so the motion vector is always 0 ((0,0)).

Returning to FIG. 10, the motion vector detector 32 of the upper layer encoding unit 23 uses the flag re as described above.
f_layer_id and ref_select_c
ode is set, the motion compensator 42 and the VLC unit 36
Is supplied to.

In the motion vector detector 32, the flags ref_layer_id and ref_selec are set.
According to t_code, not only the frame memory 31 is referred to, but also the frame memory 52 is referred to as needed, so that the motion vector is detected.

Here, the frame memory 52 is supplied with the locally decoded enlarged image of the lower layer from the resolution conversion unit 24 (FIG. 3). That is, in the resolution conversion unit 24, the locally decoded VOP of the lower layer is
However, for example, it is enlarged by a so-called interpolation filter or the like, whereby an enlarged image obtained by multiplying the VOP by FR, that is, an enlarged image having the same size as the VOP of the upper layer corresponding to the VOP of the lower layer is generated. And is supplied to the upper layer encoding unit 23. The frame memory 52 stores the enlarged image thus supplied from the resolution conversion unit 24.

Therefore, when the scaling factor FR is 1, the resolution conversion section 24 directly processes the locally decoded VOP from the lower layer coding section 25 without any processing. 23.

The motion vector detector 32 is supplied with the size data FSZ_B and the offset data FPOS_B from the lower layer encoding unit 25 and the magnification FR from the delay circuit 22 (FIG. 3). Therefore, the motion vector detector 32 uses the V when the enlarged image stored in the frame memory 52 is used as the reference image, that is, in the predictive encoding of the VOP of the upper layer.
When the VOP of the lower layer at the same time as the OP is used as the reference image (in this case, the flag ref_selec
t_code is "11" for P pictures and B
The picture is set to "00"), and the size data FSZ_B and the offset data FPOS_B corresponding to the enlarged image are multiplied by the magnification FR. Then, based on the multiplication result, the position of the enlarged image in the absolute coordinate system is recognized, and the motion vector is detected.

The motion vector detector 32 is supplied with the motion vector of the lower layer and the prediction mode, which is used in the following case.
That is, the motion vector detection unit 32, for example, flags ref_select_ for the B picture of the upper layer.
When the code is "00", the magnification FR is 1
, That is, in the case of SNR scalability (however, in this case, since the VOP of the upper layer is used for predictive coding of the upper layer, in this respect, the SN referred to here).
R scalability is different from that specified in MPEG2), and since the upper layer and the lower layer are the same image, the predictive encoding of the B picture of the upper layer is
The motion vector and prediction mode of the image at the same time in the lower layer can be used as they are. Therefore, in this case, the motion vector detection unit 32 does not perform any particular processing on the B picture of the upper layer, and directly adopts the motion vector and the prediction mode of the lower layer.

In this case, the upper layer coding unit 23
Then, from the motion vector detector 32 to the VLC device 36,
Motion vectors and prediction modes are not output (and thus not transmitted). This is because the receiving side can recognize the motion vector and prediction mode of the upper layer from the decoding result of the lower layer.

As described above, the motion vector detector 32
Detects the motion vector by using not only the VOP of the upper layer but also the enlarged image as the reference image, and further sets the prediction mode that minimizes the prediction error (or variance) as described with reference to FIG. . Also, the motion vector detector 3
2 is, for example, a flag ref_select_code
And ref_layer_id and other necessary information are set and output.

In FIG. 10, the lower layer coding unit 2
5, the flag COD indicating whether or not the macroblock forming the I or P picture in the lower layer is the skip macroblock is the motion vector detector 3
2, the VLC unit 36, and the motion compensator 42.

The detected macroblock of the motion vector is encoded in the same manner as described above, so that V
The LC unit 36 outputs a variable length code as the encoding result.

The VLC unit 36 of the upper layer encoding unit 23 is adapted to set and output the flags COD and MODB as in the case of the lower layer encoding unit 25. Here, as described above, the flag COD indicates whether the macroblock of the I or P picture is the skip macroblock, but the flag MODB indicates whether the macroblock of the B picture is the skip macroblock. It shows how.

In addition, the VLC unit 36 includes a quantization coefficient, a quantization step, a motion vector, a prediction mode,
Magnification FR, flag ref_select_code, r
The ef_layer_id, the size data FSZ_E, and the offset data FPOS_E are also supplied, and the VLC unit 36 outputs all of these data by variable length coding.

On the other hand, the macroblock in which the motion vector is detected is coded, then locally decoded as described above and stored in the frame memory 41. Then, in the motion compensator 42, in the same manner as in the motion vector detector 32, not only the locally decoded upper layer VOP stored in the frame memory 41 but also the local decoding stored in the frame memory 52. The motion compensation is performed by using the VOP of the lower layer that has been expanded as a reference image, and the predicted image is generated.

That is, the motion compensator 42 has a flag ref_select_ in addition to the motion vector and the prediction mode.
code, ref_layer_id, magnification FR, size data FSZ_B, FSZ_E, offset data F
POS_B and FPOS_E are supplied, and the motion compensator 42 sets the flag ref_select.
A reference image to be motion-compensated is recognized based on _code, ref_layer_id, and when a locally decoded VOP of the upper layer or an enlarged image is used as the reference image, its position and size in the absolute coordinate system. Is recognized based on the size data FSZ_E and the offset data FPOS_E or the size data FSZ_B and the offset data FPOS_B, and a prediction image is generated using the scaling factor FR as necessary.

Next, FIG. 13 shows a configuration example of an embodiment of a decoder for decoding the bit stream output from the encoder of FIG.

A bit stream provided from the encoder of FIG. 1 via the transmission path 5 or the recording medium 6 is supplied to this decoder. That is, the bit stream output from the encoder of FIG. 1 and transmitted via the transmission path 5 is received by a receiver (not shown) or the bit stream recorded on the recording medium 6 is reproduced (not shown). It is reproduced by the device and supplied to the demultiplexing unit 71.

In the demultiplexing unit 71, the bit stream (VS (Video Stream) described later) input thereto is received. Further, in the demultiplexing unit 71, the input bitstream is a bitstream VO # for each VO.
1, VO # 2, ... And supplied to the corresponding VOP decoding unit 72 _n . The VOP decoding unit 72 _n extracts the VO from the bit stream from the demultiplexing unit 71.
VOP (image data) and size data (VOP
size) and offset data (VOP offset) are decoded and supplied to the image reconstruction unit 73.

In the image reconstructing unit 73, the VOP decoding unit 72
_The original image is reconstructed based on the output from each of _{1 to} 72 _N. The reconstructed image is supplied to the monitor 74 and displayed, for example.

Next, FIG. 14 shows a configuration example of the VOP decoding unit 72 _n of FIG. 13 which realizes scalability.

The bitstream supplied from the demultiplexing unit 71 (FIG. 13) is input to the demultiplexing unit 91, where it is separated into an upper layer VOP bitstream and a lower layer VOP bitstream. To be done. The bit stream of the VOP of the upper layer is delayed by the delay circuit 92 for the processing time in the lower layer decoding unit 95, and then supplied to the upper layer decoding unit 93, and the bit stream of the VOP of the lower layer is lower. It is supplied to the layer decoding unit 95.

The lower layer decoding unit 95 decodes the lower layer bitstream, and the decoded image of the lower layer obtained as a result is supplied to the resolution conversion unit 94. The lower layer decoding unit 95 also receives size data FS obtained by decoding the lower layer bitstream.
Information necessary for decoding the VOP of the upper layer, such as Z_B, offset data FPOS_B, motion vector (MV), prediction mode, and flag COD, is supplied to the upper layer decoding unit 93.

In the upper layer decoding unit 93, the delay circuit 92
The upper layer bitstream supplied via
The output of the lower layer decoding unit 95 and the resolution conversion unit 94 is decoded by referring to the output as necessary, and the decoded image of the upper layer obtained as a result, the size data FSZ_
E and offset data FPOS_E are output. Further, the upper layer decoding unit 93 determines the scaling factor FR obtained by decoding the upper layer bitstream.
Is output to the resolution conversion unit 94. In the resolution conversion unit 94, the decoded image of the lower layer is converted using the magnification FR from the upper layer decoding unit 93 in the same manner as in the resolution conversion unit 24 in FIG. The enlarged image obtained by this conversion is supplied to the upper layer decoding unit 93 and is used for decoding the upper layer bit stream as described above.

Next, FIG. 15 shows a configuration example of the lower layer decoding unit 95 of FIG. Note that, in the figure, portions corresponding to those in the decoder of FIG. 39 are denoted by the same reference numerals. That is, the lower layer decoding unit 95 is basically configured similarly to the decoder of FIG.

The lower layer bit stream from the demultiplexing unit 91 is supplied to the buffer 101, where it is received and temporarily stored. The IVLC unit 102 appropriately reads out the bit stream from the buffer 101 in accordance with the processing state of the subsequent block, and performs variable length decoding on the bit stream to obtain a quantization coefficient, a motion vector, a prediction mode, and a quantization step. , Size data FSZ
_B, offset data FPOS_B, and flag C
Separate OD etc. The quantization coefficient and the quantization step are supplied to the dequantizer 103, and the motion vector and the prediction mode are the motion compensator 107 and the upper layer decoding unit 9
3 (FIG. 14). Also, size data FSZ
_B and offset data FPOS_B are supplied to the motion compensator 107, the image reconstruction unit 73 (FIG. 13), and the upper layer decoding unit 93, and the flag COD is supplied to the upper layer decoding unit 93.

In the inverse quantizer 103, the IDCT unit 104, the arithmetic unit 105, the frame memory 106, or the motion compensator 107, the inverse quantizer 38, the IDCT unit 39, and the arithmetic unit of the lower layer encoding unit 25 of FIG. 40, the frame memory 41, or the motion compensator 42, respectively, the same processing is performed to decode the VOP of the lower layer, and the image reconstructing unit 73, the upper layer decoding unit 93, and the resolution converting unit 94 ( 14).

Next, FIG. 16 shows a configuration example of the upper layer decoding unit 93 of FIG. Note that, in the figure, parts corresponding to those in FIG. 39 are denoted by the same reference numerals. That is, the upper layer decoding unit 93 is basically configured similar to the encoder of FIG. 39 except that the frame memory 112 is newly provided.

The upper layer bit stream from the demultiplexing unit 91 is passed through the buffer 101 to the IVLC unit 10.
2 is supplied. The IVLC unit 102 performs variable length decoding on the bit stream of the upper layer to obtain a quantization coefficient,
Motion vector, prediction mode, quantization step, size data FSZ_E, offset data FPOS_E, scaling factor FR, flags ref_layer_id, ref_se
Lect_code, COD, MODB, etc. are separated. The quantization coefficient and the quantization step are supplied to the dequantizer 103, and the motion vector and the prediction mode are supplied to the motion compensator 107, as in the case of FIG. Further, the size data FSZ_E and the offset data FPOS_E are supplied to the motion compensator 107 and the image reconstruction unit 73 (FIG. 13), and the flags COD, MO are supplied.
DB, ref_layer_id, and ref_se
The lect_code is supplied to the motion compensator 107. Further, the magnification FR is supplied to the motion compensator 107 and the resolution converter 94 (FIG. 14).

In addition to the above-described data, the motion compensator 107 receives the motion vector of the lower layer, the flag COD, and the size data F from the lower layer decoding unit 95 (FIG. 14).
SZ_B and offset data FPOS_B are supplied. In addition, the frame memory 11
An enlarged image is supplied to 2 from the resolution conversion unit 94.

Inverse quantizer 103, IDCT device 104, calculator 105, frame memory 106, motion compensator 10
7, or in the frame memory 112, the inverse quantizer 38, the IDCT device 39, and the inverse quantizer 38 of the upper layer encoding unit 23 of FIG.
By performing the same processing as in the arithmetic unit 40, the frame memory 41, the motion compensator 42, or the frame memory 52, the VOP of the upper layer is decoded and supplied to the image reconstructing unit 73.

Here, a VO having the upper layer decoding unit 93 and the lower layer decoding unit 95 configured as described above.
In the P decoding unit 72 _n , the decoded image of the upper layer, the size data FSZ_E, and the offset data FPOS_E (hereinafter, including all of them, as appropriate,
Upper layer data), a decoded image of the upper layer of the lower layer, and size data FSZ_
B and offset data FPOS_B (hereinafter, all of them are appropriately included and referred to as lower layer data)
However, the image reconstruction unit 73 is configured to reconstruct an image from the upper layer data or the lower layer data in the following manner, for example.

That is, for example, in the case where the first spatial scalability (FIG. 4) is performed (when the entire input VOP is the upper layer and the reduced VOP is the lower layer). When both the lower layer data and the upper layer data are decoded, the image reconstruction unit 73 generates a decoded image (VOP) of the upper layer having a size corresponding to the size data FSZ_E based on only the upper layer data. , At the position indicated by the offset data FPOS_E. Further, for example, when an error occurs in the bit stream of the upper layer, and the monitor 74 supports only low resolution images, when only the lower layer data is decoded, the image reconstructing unit 73. Arranges the decoded image (VOP) of the upper layer having the size corresponding to the size data FSZ_B at the position indicated by the offset data FPOS_B based on only the lower layer data.

In addition, for example, when the second spatial scalability (FIG. 5) is performed (a part of the input VOP is set as the upper layer, and a reduction of the entire VOP is set as the lower layer. In the case), when both the lower layer data and the upper layer data are decoded, the image reconstruction unit 73 determines that the size data F
The decoded image of the lower layer of the size corresponding to SZ_B is
The image is enlarged according to the magnification FR, and the enlarged image is generated. Further, the image reconstructing unit 73 uses the offset data F
POS_B is multiplied by FR, and the enlarged image is arranged at a position corresponding to the value obtained as a result. Then, the image reconstruction unit 7
3 arranges the decoded image of the upper layer having the size corresponding to the size data FSZ_E at the position indicated by the offset data FPOS_E.

In this case, the part of the decoded image in the upper layer is displayed with a higher resolution than the other parts.

When the decoded image of the upper layer is arranged, the decoded image and the enlarged image are combined.

Although not shown in FIG. 14 (FIG. 13), in addition to the above-mentioned data, the upper layer decoding unit 93 (VOP decoding unit 72 _n ) to the image reconstruction unit 73
The magnification FR is also supplied, and the image reconstructing unit 73 is configured to generate an enlarged image using this.

On the other hand, in the case where the second spatial scalability is performed, when only the lower layer data is decoded, the image is reconstructed in the same manner as in the case where the first spatial scalability is performed. .

Furthermore, when the third spatial scalability (FIGS. 6 and 7) is performed (for each object forming the input VOP, the entire object (object) is set as an upper layer, and the entire object is In the case where the thinned layer is used as the lower layer), the image is reconstructed in the same manner as in the case where the second spatial scalability described above is performed.

As described above, the offset data FPO
In S_B and FPOS_E, the corresponding pixels forming the enlarged image of the lower layer and the image of the upper layer are arranged at the same position in the absolute coordinate system. With this configuration, an accurate image (without positional deviation) can be obtained.

[0167] Next, regarding the syntax of the coded bit stream output by the encoder of Fig. 1, for example, MPEG
An example will be described using the Video Verification Model (Version 6.0) of 4 standards (hereinafter, appropriately referred to as VM6.0).

FIG. 17 shows the structure of a coded bitstream in VM6.0.

The coded bit stream is VS (Video
Session Class), and each VS has 1
It is composed of the above VO (Video Object Class). The VO is one or more VOLs (Video Object Layer C
lass) (V of 1 when the image is not layered)
If the image is layered, it is composed of the number of VOLs corresponding to the number of layers) and VOL is VOP (Vi
deo Object Plane Class).

The VS is an image sequence and corresponds to, for example, one program or movie.

FIG. 18 or FIG. 19 shows the syntax of VS or VO, respectively. A VO is a bitstream corresponding to a sequence of an entire image or a part (object) of an image, and thus a VS is composed of a set of such sequences (so VS is, for example, a program Equivalent to).

FIG. 20 shows the syntax of the VOL.

The VOL is a class for scalability as described above, and is identified by the number indicated by video_object_layer_id. That is, for example, video_object_layer_id for the VOL of the lower layer is set to 0, and, for example, vi for the VOL of the upper layer.
deo_object_layer_id is set to 1. Note that, as described above, the number of scalable layers is not limited to 2, and can be any number including 1 or 3 or more.

For each VOL, video_object_l is used to determine whether it is the entire image or a part of the image.
Identified by ayer_shape. This video_object_layer_sha
pe is a flag indicating the shape of the VOL, and is set as follows, for example.

That is, when the VOL has a rectangular shape, video_object_layer_shape is set to, for example, "00". Also, when the VOL has a shape of an area extracted by a hard key (a binary signal that takes one of 0 and 1), video_ob
The ject_layer_shape is set to "01", for example. Further, when the VOL has a shape of an area extracted by a soft key (a signal that can take a continuous value (Gray-Scale) in the range of 0 to 1) (combined by using the soft key). Video_object_layer
_shape is set to “10”, for example.

Here, video_object_layer_shape is "0".
“0” is defined when the VOL has a rectangular shape and the position and size of the VOL in the absolute coordinate form do not change with time, that is, are constant. In this case, the sizes (horizontal length and vertical length) are video_object_layer_width and video_object_lay.
Indicated by er_height. video_object_layer_widt
Both h and video_object_layer_height are fixed-length flags of 10 bits, and when video_object_layer_shape is "00", it is transmitted only once at the beginning (this is as described above when video_object_layer_shape is "00"). , The size of the VOL in the absolute coordinate system is constant).

Whether the VOL is a lower layer or an upper layer is indicated by the scalability which is a 1-bit flag. If the VOL is a lower layer, the scalability is set to 1, for example, and otherwise the scalability is set to 0, for example.

Further, when a VOL uses an image in a VOL other than itself as a reference image, the VOL to which the reference image belongs is represented by ref_layer_id as described above. Note that ref_layer_id is transmitted only for the upper layer.

Further, in FIG. 20, hor_sampling_fac
tor_n and hor_sampling_factor_m are VOPs of the lower layer
Value corresponding to the horizontal length of the VOP of the upper layer
The respective values corresponding to the horizontal length of are shown. Therefore, the horizontal length of the upper layer with respect to the lower layer (magnification of the horizontal resolution) is calculated by the formula hor_sampling_facto
It is given by r_n / hor_sampling_factor_m.

Further, in FIG. 20, ver_sampling_f
actor_n and ver_sampling_factor_m are lower layer VO
A value corresponding to the vertical length of P and the VO of the upper layer
The values corresponding to the vertical length of P are shown. Therefore, the vertical length (magnification of the vertical resolution) of the upper layer with respect to the lower layer is calculated by the formula ver_sampling_facto
It is given by r_n / ver_sampling_factor_m.

Next, FIG. 21 shows a VOP (Video Object P
(lane class) syntax is shown.

The VOP size (horizontal and vertical lengths) is represented by VOP_width and VOP_height having a fixed length of 10 bits, for example. Further, the position of the VOP in the absolute coordinate system is, for example, VOP_horizontal_spatial_mc_re having a fixed length of 10 bits.
It is represented by f and VOP_vertical_mc_ref. Note that VOP_width
Alternatively, VOP_height represents the horizontal or vertical length of the VOP, and these correspond to the size data FSZ_B and FSZ_E described above. Also, VOP_horizo
ntal_spatial_mc_ref or VOP_vertical_mc_ref is V
Represents horizontal or vertical coordinates (x or y coordinates) of OP, which correspond to the offset data FPOS_B and FPOS_E described above.

VOP_width, VOP_height, VOP_horizontal_
spatial_mc_ref and VOP_vertical_mc_ref are video
It is transmitted only when _object_layer_shape is other than "00". That is, when the video_object_layer_shape is “00”, as described above, the size and position of the VOP are constant, so VOP_width, VOP_height, VOP_h.
orizontal_spatial_mc_ref, and VOP_vertical_mc_re
f need not be transmitted. In this case, on the receiving side, VO
20. P is arranged such that its upper left apex coincides with, for example, the origin of the absolute coordinate system, and its size is as shown in FIG.
Video_object_layer_width and video_obje described in
Recognized by ct_layer_height.

In FIG. 21, ref_select_code represents an image used as a reference image as described in FIG. 17, and is defined in the VOP syntax.

By the way, in VM6.0, each VOP (Video Objec
(t Plane: Corresponding to conventional Frame) is displayed at the modulo
_time_base and VOP_time_increment (FIG. 21) determine as follows.

That is, modulo_time_base represents the time on the local time axis of the encoder with an accuracy of 1 second (1000 ms (millisecond)). modulo_time_base is expressed by a marker transmitted in the VOP header, and is composed of a required number of "1" s and 1 "0" s. modulo_tim
The number of "1" s that make up the e_base is the modulo that was last encoded (back to the present and most recently) (immediately before).
Sync point indicated by _time_base (1 second precision time)
Represents the cumulative time from. That is, when modulo_time_base is, for example, “0”, modu that was encoded / decoded immediately before
Indicates that the cumulative time from the sync point indicated by lo_time_base is 0 seconds. Further, when modulo_time_base is, for example, “10”, the mo coded / decoded immediately before.
Indicates that the cumulative time from the sync point indicated by dulo_time_base is 1 second. In addition, modulo_time_base
However, for example, “110” indicates that the cumulative time from the synchronization point indicated by modulo_time_base encoded / decoded immediately before is 2 seconds. As mentioned above, modu
The number of "1" in lo_time_base is the number of seconds from the sync point indicated by modulo_time_base that was encoded / decoded immediately before.

In VM6.0, modulo_time_base is "This value represents the local time base a
t the one second resolution unit (1000 millisecond
s) .It is represented as a marker transmitted in th
e VOP header. The number of consecutive "1" followe
d by a "0" indicates the number of seconds has ela
psed since the synchronization point marked by the
last encoded / decoded modulo_time_base. "

VOP_time_increment represents the time on the local time axis of the encoder with an accuracy of 1 ms. VM
In 6.0, VOP_time_increment is modulo_time_base encoded / decoded immediately before for I-VOP and P-VOP.
Indicates the time from the synchronization point indicated by, and for B-VOP, the relative time from the immediately preceding encoded / decoded I-VOP or P-VOP.

In VM6.0, regarding VOP_time_increment, "This value represents the local time bas
e in the units of milliseconds.For I and P-VOP's
thisvalue is the absolute VOP_time_increment from
the synchronization pointmarked by the last modulo
_time_base.For the B-VOP's this value is the rela
tive VOP_time_increment from the last encoded / deco
ded I- or P-VOP. "

Then, in VM6.0, "At the encoder, t
he following formula are used to determine the abso
lute and relative VOP_time_increments for I / P-VOP '
s and B-VOP's, respectively. "

That is, in the encoder, it is stipulated that the display time of each of the I-VOP and P-VOP and the B-VOP is coded using the following formula.

T _{GTB (n)} = n × 1000 ms + t _EST t _AVTI = t _{ETB (I / P)} −t _{GTB (n)} t _RVTI = t _{ETB (B)} −t _{ETB (I / P)} (1 However, in Expression (1), t _{GTB (n)} represents the time of the synchronization point indicated by the nth encoded modulo_time_base (second precision as described above), and t _EST is in the encoder. It represents the VO encoding start time (absolute time when the VO encoding was started). Also, t _AVTI represents VOP_time_increment for I-VOP or P-VOP, and t _{AVTI
ETB (I / P)} is the encoding start time of I-VOP or P-VOP in the encoder (absolute time when the encoding of VOP was started)
Represents Further, t _RVTI is VOP_time_ for B-VOP.
represents the increment and t _{ETB (B)} is the BV at the encoder
Indicates the OP start time of OP.

In VM6.0, t in equation (1)
_{GTB (n)} , t _EST , t _AVTI , t _{ETB (I / P)} , t _RVTI , t
Regarding _{ETB (B)} , "t _{GTB (n)} is the encoder time base
marked by the nth encoded modulo_time_base, t _EST
is the encoder time base start time, t _AVTI is the
absolute VOP_time_increment for the I or P-VOP, t
_{ETB (I / P )} is the encoder time base at the start of
the encoding of the I or P-VOP, t _RVTI is the _relat
ive VOP_time_increment for the B-VOP, and t _{ETB (B)}
is the encoder time base at the start of the encod
ing of the B-VOP. "

Further, in VM6.0, "At the decoder, the
following formula are used to determine the recov
ered time base of the I / P-VOP's and B-VOP's, respe
ctively: ”is described.

That is, on the decoder side, it is specified that the display times of I-VOP and P-VOP and B-VOP are decoded using the following formula.

T _{GTB (n)} = n × 1000 ms + t _DST t _{DTB (I / P)} = t _AVTI + t _{GTB (n)} t _{DTB (B)} = t _RVTI + t _{DTB (I / P)} (2) In the equation (2), t _{GTB (n)} represents the time of the sync point indicated by the nth decoded modulo_time_base, and t _DST is the decoding start time of the VO in the decoder (the decoding of VO is started. Absolute time). Also, t
_{DTB (I / P)} represents the decoding start time of the I-VOP or P-VOP in the decoder, and t _AVTI represents the VOP_time_increment for the I-VOP or P-VOP. Further, t _{DTB (B)} represents the decoding start time of the B-VOP in the decoder (the absolute time when the decoding of the VOP was started), and t _{RVT I} represents the VOP_time_increment of the B-VOP.

In VM6.0, t in equation (2)
_{GTB (n)} , t _DST , t _{DTB (I / P)} , t _AVTI , t _{DTB (B)} , t
About _RVTI , "t _{GTB (n)} is the encoding time base"
marked bythe nth decoded modulo_time_base, t _DST is
the decoding time base start time, t _{DTB (I / P)} is t
he decoding time base at the start of the decoding
of the I or P-VOP, t _AVTI is the decoding absolute
VOP_time_increment for the I or P-VOP, t _{DTB (B)} is
the decoding time base at the start of the decodin
g of the B-VOP, and t _RVTI is the decoded relative
VOP_time_increment for the B-VOP. "

FIG. 22 shows modulo based on the above definition.
It is a figure showing the relation between _time_base and VOP_time_increment.

In FIG. 22, VO is I1 (I-VO
P), B2 (B-VOP), B3, P4 (P-VO
P), B5, P6, ... VOP sequences. Now, assuming that the VO encoding / decoding start time (absolute time) is t0, modulo_time_base represents the elapsed time from time t0 with 1 second precision, so t0 +
It represents a time (synchronization point) of 1 second, t0 + 2 seconds, .... In FIG. 22, the display order is I1, B2, B.
3, P4, B5, P6, ..., The encoding / decoding order is I1, P4, B2, B3, P6 ,.

In FIG. 22, (FIGS. 25 to 28 described later,
Also in FIG. 33), VO for each VOP
A number that encloses P_time_increment in a rectangle (unit is ms)
, And the switching of the synchronization points indicated by modulo_time_base is indicated by the ▼ mark. Therefore, FIG.
Then, for I1, B2, B3, P4, B5, P6
VOP_time_increment is 350ms, 400ms, 80
The synchronization points are 0 ms, 550 ms, 400 ms, and 350 ms, respectively, and the synchronization points are switched at P4 and P6.

Now, in FIG. 22, VOP_time_i of I1
Since ncrement is 350 ms, the encoding / decoding time of I1 is the time 350 ms after the synchronization point indicated by modulo_time_base encoded / decoded immediately before.
Immediately after the start of encoding / decoding, the start time (encoding / decoding start time) t0 becomes the synchronization point, so the encoding / decoding time of I1 is 35 from the encoding / decoding start time t0.
This means that the time t0 + 350 ms after 0 ms.

Since the coding / decoding time of B2 or B3 is the time when VOP_time_increment has elapsed from the I / VOP or P-VOP coded / decoded immediately before,
In the present case, the last encoded / decoded I1 encoded /
400 ms or 80 from decoding time t0 + 350 ms
Time t0 + 750 ms after 0 ms or t0 + 1200
It will be ms, respectively.

Next, regarding P4, in P4,
The sync point indicated by modulo_time_base has been switched, so the sync point is time t0 + 1 seconds. As a result, the encoding / decoding time of P4 is the time (t0 + 1) seconds +550 ms 550 ms after the time t0 + 1 seconds.

The coding / decoding time of B5 is VOP_time_increme from the I / VOP or P-VOP coded / decoded immediately before.
Since the time has passed by nt, in the present case, the encoding / decoding time (t0 + 1) of the last encoded / decoded P4
Time (t0 + 1) after 400 ms from second +550 ms
This means seconds +950 ms.

Next, regarding P6, in P6,
The sync point indicated by modulo_time_base has been switched, so the sync point is time t0 + 2 seconds. As a result, the encoding / decoding time of P6 is the time (t0 + 2) seconds + 350 ms 350 ms after the time t0 + 2 seconds.

Incidentally, in VM6.0, the switching of the synchronization points indicated by modulo_time_base is performed by I-VOP and P-
Only allowed for VOP and not for B-VOP.

In VM6.0, VOP_time_incremen
For I-VOP and P-VOP, t represents the time from the sync point indicated by modulo_time_base encoded / decoded immediately before, whereas for B-VOP only, encoded / decoded immediately before. The reason why it is supposed to represent the relative time from the I-VOP or P-VOP is mainly as follows. That is, B-VOPs are I-Vs that sandwich the B-VOP in display order.
Since OP or P-VOP is predictively coded as a reference image, the weight for I-VOP or P-VOP used as a reference image at the time of predictive coding, from B-VOP, I- sandwiching it-
To determine based on the temporal distance to VOP or P-VOP, the temporal distance is set to VOP_time_in for B-VOP.
Because it was crement.

By the way, the above-mentioned VOP_time_incr of VM6.0
The definition of ement causes inconvenience. That is, in FIG. 22, the VOP_time_increment for the B-VOP is the B-VOP.
Is not the relative time from the I-VOP or P-VOP coded / decoded immediately before, but the relative time from the I-VOP or P-VOP displayed immediately before. This is for the following reason. That is, for example, when attention is paid to B2 or B3, the I-VOP or P-VOP coded / decoded immediately before that.
Is P4 in the encoding / decoding order described above.
Therefore, for B-VOP, VOP_time_increment is
-If the relative time from the I-VOP or P-VOP coded / decoded immediately before VOP is represented, VO for B2 and B3
P_time_increment represents the relative time from the encoding / decoding time of P4, and has a negative value.

[0209] On the other hand, according to the MPEG4 standard, VOP_time_incremen
Since t is 10 bits, and if it takes only a value of 0 or more, a value in the range of 0 to 1023 can be expressed. Therefore, the position between adjacent sync points can be temporally determined. It can be expressed in units of 1 ms with the synchronization point located in front (to the left in FIG. 22) as a reference.

However, VOP_time_increment is 0
If not only the above values but also negative values are allowed, for example, the positions between adjacent sync points are expressed with reference to the sync point located earlier in time, or in terms of time. Since it is expressed with a synchronization point located later as a reference, the process of obtaining the VOP encoding time and VOP decoding time becomes complicated.

Therefore, in VM6.0, as described above, VOP
_time_increment says `` This value represents the loca
l time base in the units of milliseconds.For I a
ndP-VOP's this value is the absolute VOP_time_incr
ement from the synchronization point marked by the
last modulo_time_base.For the B-VOP's this value
is the relative VOP_time_increment from the last
is defined as encoded / decoded I- or P-VOP.
The last sentence “For the B-VOP's this value is the relat
ive VOP_time_increment from the last encoded / decod
ed I- or P-VOP ”is“ For the B-VOP's this value i
s the relative VOP_time_increment from the last di
splayed I- or P-VOP ”, so that the VOP_time_increment is not the relative time from the last encoded / decoded I-VOP or P-VOP, but the last displayed I- It should be defined to represent the relative time from the VOP or P-VOP.

By defining VOP_time_increment in this way, the I / PV having a display time that is earlier than the B-VOP is used as the basis for calculating the encoding / decoding time for the B-VOP.
The OP (I-VOP or P-VOP) display time comes, so B-VOP
VOP_time_increment is the I-VO it references
Unless P is displayed before its B-VOP, it will always have a positive value, and therefore the I / P-VOP's VOP_time_in
crement will always take a positive value.

Further, in FIG. 22, the definition of VM6.0 is further defined.
Change to modulo_time_base and VOP_time_increment
The time represented by is not the encoding / decoding time,
It is supposed to be the display time of the VOP. That is, in FIG. 22, V
Considering the absolute time on the OP sequence, equation (1)
At t_{EST (I / P)}And t in equation (2)
_{DTB (I / P)}Is the sequence on which the I or P-VOP is located.
The time is represented by t in equation (1)._{EST (B)}And in equation (2)
T_{DTB (B)}Is an absolute on the sequence where the B-VOP is located
Each time is shown.

Next, in VM6.0, the coding start time t _EST (the encoder timebase start time) in equation (1) is not coded, and the coding start time t _EST and the display time of each VOP ( Modulo_time_base and VOP_ as difference information with the absolute time that represents the position of each VOP on the VOP sequence)
time_increment is encoded. Therefore, on the decoder side, modulo_time_base and VOP_time_increment can be used to determine the relative time relationship between each VOP, but the absolute display time of each VOP, that is, each VOP
It is not possible to define where in P's sequence it is. Therefore, modulo_time_base and VO
P_time_increment alone cannot access the middle of the bitstream, that is, random access cannot be performed.

On the other hand, if the encoding start time t _EST is simply encoded, the decoder can use it to decode the absolute time of each VOP, but the encoding is always performed from the beginning of the encoded bit stream. Start time t _EST and modulo_time_base and VOP_time_i which are relative time information of each VOP.
It is necessary to accumulate the ncrement and accumulate it while managing the absolute time, which is troublesome and cannot perform efficient random access.

Therefore, in the present embodiment, VM6.
Within the structure (layer) of the coded bitstream of 0, VOP
Introduces a layer that encodes absolute time on the sequence (this layer is a layer of an encoded bitstream, not a layer that realizes scalability (the above-described lower layer and upper layer)). This layer is a layer of a coded bitstream that can be inserted not only at the head of the coded bitstream but also at an appropriate position.

[0217] Here, as the hierarchy, for example, MPEG
Introduces the one specified in the same way as the GOP (Group of Picture) layer used in 1/2. As a result, compared to the case of using a unique encoded stream layer for MPEG4, MPEG4
4 and MPEG1 / 2 compatibility (Compatibility)
Can be increased. This newly introduced layer is here referred to as GOV (or GVOP) (Group Of Video O
bject Plane).

FIG. 23 shows a structural example of a coded bitstream in which a GOV layer for coding absolute time on a VOP sequence is introduced.

The GOV layer is defined between the VOL layer and the VOP layer so that the GOV layer can be inserted not only at the head of the bitstream but also at an arbitrary position of the encoded bitstream.

As a result, a certain VOL # 0 becomes VOP # 0, VOP #
1, ..., VOP # n, VOP # (n + 1), ..., VOP # m
When it is composed of VOP sequence, GOV layer is
It can be inserted immediately before VOP # (n + 1) as well as immediately before the first VOP # 0. Therefore, in the encoder, the GOV layer can be inserted, for example, at a position in the encoded stream at which random access is desired. Therefore, by inserting the GOV layer, a sequence of VOPs constituting a certain VOL is sequenced. Will be coded by being divided into a plurality of groups (hereinafter appropriately referred to as GOV) by the GOV layer.

The syntax of the GOV layer is defined as shown in FIG. 24, for example.

As shown in the figure, the GOV layer has a group start code (group_start_code) and a time code (ti
me_code), closed GOP (closed_gop), broken link (broken_link), next start code (nex)
t_start_code ()) is sequentially arranged and configured.

Next, the semantics of the GOV layer (Semantic
s) will be described. The semantics of the GOV layer are basically the same as the GOP layer of MPEG2, so
For parts that are not particularly described, the MPEG2 Video standard (ISO / I
See EC13818-2).

Group_start_code is 000001B8 (hexadecimal number) and indicates the start position of the GOV.

The time_code is 1-bit drop_frame_flag, 5-bit time_code_hours, 6 as shown in Table 1.
Bit time_code_minutes, 1 bit marker_bit,
6-bit time_code_seconds and 6-bit time_
It consists of a total of 25 bits of code_pictures.

[0226]

[Table 1]

[0227] time_code is IEC standard publication
461 `` time and control codes for vi
equivalent to "deo tape recorders". Here, since there is no concept of video frame rate (Video Frame Rate) in MPEG4 (hence, VOP can be displayed at any time), here time_code is drop frame mode (drop_frame_mode). The value is fixed to 0, for example, without using the drop_frame_flag indicating whether or not it is described. For the same reason, time_code_picture
The value of s is fixed to 0, for example, without using s. Therefore, here, time_code represents t, which represents the time unit of time.
ime_code_hours, time_code_minu representing minutes of the time
The time at the beginning of the GOV is represented by tes and time_code_seconds representing the unit of time seconds. As a result, the GOV layer ti
me_code (encoding start second precision absolute time) is second precision,
The leading time, that is, the absolute time on the VOP sequence at which the coding of the GOV layer is started will be expressed. Therefore, in the present embodiment, a time (time) (here, millisecond) having a precision finer than seconds is set for each VOP.

The marker_bit of time_code is set to 1 so that 23 or more 0s do not continue in the coded bit stream.

Closed_gop is the MPEG2 Video standard (ISO / IEC 1
3818-2) I, P in the definition of close_gop definition,
Or B-pictures are replaced with I-VOPs, P-VOPs, or B-VOPs, respectively. Therefore, a B-VOP in a GOV is not only a VOP that composes the GOV, but also another GOV
It indicates whether or not the VOP forming the is encoded as a reference image. Here, the MPEG2 Video standard (ISO / IEC
Regarding the definition of close_gop in 13818-2), the following is a sentence that has been replaced as described above.

This is a one-bit flag which indicates
the nature of the predictions used in the first c
onsecutive B-VOPs (if any) immediately following t
he first coded I-VOP following the group of plane
header.The closed_gop isset to 1 to indicate that
these B-VOPs have been encoded using only backwar
d prediction or intra coding.This bit is provided
for use during anyediting which occurs after enco
ding. If the previous pictures have been removed by
editing, broken_link may be set to 1 so that a de
coder may avoid displaying these B-VOPs following
the first I-VOP following the group of plane heade
r. However if the closed_gop bit is set to 1, then
theeditor may choose not to set the broken_link b
it as these B-VOPs can becorrectly decoded.

[0231] The broken_link also conforms to the MPEG2 Video standard (ISO / IEC
13818-2) regarding the description of broken_link, closed_link
It means the same replacement as in gop, and therefore represents whether the head B-VOP of GOV can be reproduced correctly. Here, the following is a sentence in which the definition of broken_link in the MPEG2 Video standard (ISO / IEC 13818-2) is replaced as described above.

This is a one-bit flag which shall be
set to 0 during encoding.It isset to 1 to indicat
e that the first consecutive B-VOPs (if any) immed
iately following the first coded I-VOP following t
he group of plane headermay not be correctly decod
ed because the reference frame which is used for p
rediction is not available (because of the action
of editing) .A decoder may use this flag to avoid
displaying frames that cannot be correctly decode
d.

Next_start_code () gives the position of the beginning of the next GOV.

By introducing the GOV layer as described above and setting the GOV encoding, the absolute time on the GOV sequence (hereinafter, appropriately referred to as the encoding start absolute time) is set to the GOV time code time_code. . Further, as described above, the time_code of the GOV layer is the second precision, so here, the VOP of each VOP is
The finer precision part of the absolute time on the sequence is set for each VOP.

That is, FIG. 25 shows time_code, modulo_time_base and VOP_tim when the GOV layer of FIG. 24 is introduced.
It shows the relationship with e_increment.

In FIG. 25, GOV is
I1, B2, B3, P4, B5, P6 are arranged in the display order.

[0237] Now, for example, assuming that the absolute start time of encoding of GOV is 0h: 12m: 35sec: 350msec (0: 12: 35: 350msec), the time_code of GOV is as described above.
Since it is second precision (second unit), it is set to 0h: 12m: 35sec (time
time_code_hours, time_code_minute that compose _code
s or time_code_seconds is 0, 12, or 35, respectively). On the other hand, the absolute time on the VOP sequence of I1 (the absolute time on the VOP sequence before encoding (or after decoding) VS including GOV in FIG. 25) (this is V
It corresponds to the time when I1 is displayed when the OP sequence is displayed, so it will be referred to as the display time hereinafter).
However, for example, when it is 0h: 12m: 35sec: 350msec, 350ms which is a precision finer than the second precision of the display time is
As set by the VOP_time_increment of the I-VOP for I1 and encoded (VOP_time_incre for I1
ment = 350 and encoded), VOP_time_in
Change the semantics of crement.

That is, in FIG. 25, the VOP_time_increment of the first I-VOP (I1) in the display order of the GOV is ti of the GOV.
It is the difference between me_code and the display time of the I-VOP. Therefore,
The time represented by time_code with the second precision is the first synchronization point of GOV (here, the point representing the second precision time).

In FIG. 25, VOPs B2, B3, P4, B5, P6 which are VOPs arranged after the second GOV.
The semantics of VOP_time_increment for is similar to that when the definition of VM6.0 is changed as described in FIG.

Therefore, in FIG. 25, B2 or B3
The display time of is the time when VOP_time_increment has elapsed from the display time of the I-VOP or P-VOP displayed immediately before, so in this case, the display time of I1 displayed immediately before
From 0h: 12m: 35s + 350ms, time 0h: 12 after 400ms or 800ms
It becomes m: 35s: 750ms or 0h: 12m: 36s: 200ms, respectively.

Next, regarding P4, in P4,
The sync point indicated by modulo_time_base has been switched, so the sync point is 0h: 12m: 36s, which is one second after the time 0h: 12m: 35s. As a result, the display time of P4 is
It means 550ms after time 0h: 12m: 36s and time 0h: 12m: 36: 550ms.

The display time of B5 is the I-displayed immediately before.
Since it is the time when VOP_time_increment has passed from the VOP or P-VOP, in the present case, the P displayed immediately before is displayed.
4 display time 0h: 12m: 36: 550ms, time 400h later 0h: 12
It will be m: 36s: 950ms.

Regarding P6, in P6, the synchronization point indicated by modulo_time_base is switched, and therefore the synchronization point is time 0h: 12m: 35s + 2 seconds, that is, 0h: 12m: 37s. As a result, the display time of P6 is
It means that the time is 0h: 12m: 37s: 350ms after 350ms from the time 0h: 12m: 37s.

Next, in FIG. 26, the top VOP is B in the display order.
-The time_code for GOV when VOP is set,
It shows the relationship with modulo_time_base and VOP_time_increment.

In FIG. 26, GOV is
In display order, B0, I1, B2, B3, P4, B5, P6
Are arranged and configured. That is, in FIG.
In 5, the GOV is constructed by adding B0 before I1.

In this case, the VOP for B0 at the beginning of GOV
If _time_increment is defined based on the display time of the I / P-VOP forming the GOV, that is, for example,
If the display time of I1 is set as a reference, the value becomes negative, which is inconvenient as described above.

Therefore, in the GOV, the B-VOP displayed before the I-VOP (the B-VOP displayed prior to the I-VOP displayed first in the GOV) Change the semantics of VOP_time_increment as follows.

That is, VOP_time_in of such a B-VOP
crement is the difference value between the time of time_code of GOV and the display time of B-VOP. In this case, as shown in FIG.
The display time of 0 is, for example, 0h: 12m: 35s: 200ms, and GO
When the time_code of V is 0h: 12m: 35s, for example,
V0_time_increment of B0 is 350ms (= 0h: 12m: 35s: 20
0ms−0h: 12m: 35s). By doing this, VOP
_time_increment is always a positive value.

[0249] Due to the two changes in the semantics of VOP_time_increment as described above, GOV time_co
de and VOP modulo_time_base and VOP_time_incremen
can be related to t, and in addition, each V
The absolute time (display time) when the OP is displayed can be specified.

Next, FIG. 27 shows GOV when the interval between the display time of the I-VOP and the predicted display time of the B-VOP is longer than 1 second (more precisely, 1.023 seconds). About time_code and modulo_time_base and VOP_time_in
It shows the relationship with crement.

In FIG. 27, GOV is I in display order.
1, B2, B3, B4, P6 are sequentially arranged, and B4 is displayed at a time that is one second after the display time of I1, which is the I-VOP displayed immediately before. Has been done.

In this case, even if an attempt is made to encode the display time of B4 by VOP_time_increment whose semantics have been changed as described above, VOP_time_increment is 10 bits as described above, so it can only express up to 1023 and 1.023 seconds. It cannot express a longer time.
Therefore, the semantics of VOP_time_increment are further changed, and the semantics of modulo_time_base are also changed so that such a case can be dealt with.

Here, for example, either of the following first or second method is adopted.

That is, according to the first method, the time between the display time of the I / P-VOP and the predicted display time of the B-VOP is calculated with millisecond precision, and the time is calculated in seconds. Up to the unit, it is expressed by modulo_time_base, and the precision of the remaining milliseconds is expressed by VOP_time_increment.

In the case shown in FIG. 27, according to the first method, modulo_time_base and VOP_time_incre
time_code for GOV when ment is encoded,
FIG. 28 shows the relationship between modulo_time_base and VOP_time_increment.

That is, in the first method, modulo_time_base
Is allowed not only for I-VOP and P-VOP but also for B-VOP. And modulo_ added to B-VOP
It is assumed that time_base does not indicate switching of synchronization points, but rather carries forward by seconds from the display time of the I / P-VOP displayed immediately before.

Further, in the first method, the time after the carry-up in seconds from the display time of the I / P-VOP displayed immediately before, which is indicated by modulo_time_base added to the B-VOP, is set to the B-VOP. -The value subtracted from the display time of VOP is the V
Set as OP_time_increment.

Therefore, according to the first method, in FIG. 27, for example, the display time of I1 is set to 0h: 12m: 35s: 350ms, and the display time of B4 is set to 0h: 12m: 36s: 550ms. Then, the difference between the display times of I1 and B4 is 12 seconds or more.
Since it is 00 msec, modulo_time_base (indicated by ▼ in FIG. 28) indicating a carry in seconds from the display time of I1 displayed immediately before is added to B4, as shown in FIG. Specifically, modu added to B4
lo_time_base is set to “10”, which represents a carry of 1 second, which is a value of 1 second of 1200 ms. And V4_ of B4
As shown in FIG. 28, time_increment is a value that is less than 1 second of the difference between the display times of I1 and B4 (from the display time of B4, the I / P-VOP displayed immediately before that is indicated by its modulo_time_base). Which is a value obtained by subtracting the time after the advance in seconds from the display time of a certain I1).
It is said that

Modulo_t according to the first method as described above
On the encoder side, the processing for ime_base and VOP_time_increment is performed by the VL shown in FIGS. 9 and 10, for example.
On the decoder side in the C unit 36, for example, in the IVLC unit 102 shown in FIGS. 15 and 16, respectively.

Therefore, first, referring to the flowchart of FIG. 29, modulo_time_b of the I / P-VOP performed by the VLC unit 36.
The processing related to ase and VOP_time_increment will be described.

The VLC unit 36 changes the sequence of VOP to GOV.
It is designed to be processed separately. In addition, GO
V is configured to include at least one intra-coded VOP.

When the GOV is received, the VLC unit 36 sets, for example, the reception time of the GOV as the absolute start time of encoding of the GOV, up to the second precision of the absolute start time of encoding (the absolute start of encoding up to the second digit). Time) is encoded as time_code and included in the encoded bitstream. Then V
Each time the LC unit 36 receives an I / P-VOP forming a GOV, the LC unit 36 sets the I / P-VOP as the target I / P-VOP and follows the flowchart of FIG. 29 to follow the modulo_time_base of the target I / P-VOP.
And VOP_time_increment are obtained and encoded.

That is, in the VLC unit 36, first, in step S1, modulo_time_base is set to 0B (B is 2).
(Representing a decimal number) is set and VOP_time_incre
ment is set to 0, modulo_time_base
And VOP_time_increment are reset.

Then, the processing proceeds to step S2, and the attention I / P-VO
Among the GOVs that P is processing (GOVs to be processed),
It is determined whether it is the first I-VOP displayed (First I-VOP). At step S2, the target I / P-VOP
Is determined to be the first I-VOP to be displayed among the GOVs to be processed, the process proceeds to step S4, and the GOVs to be processed
Time_code of the target I / P-VOP (here, the target GOV
, The difference between the display time of the first displayed I-VOP) and the second precision, that is, the difference between the time_code and the second digit of the display time of the I / P-VOP of interest is calculated. After being set to D, the process proceeds to step S5.

Also, in step S2, the target I / P-VO
If it is determined that P is not the first I-VOP displayed in the GOVs to be processed, the process proceeds to step S3, and the target I / P-VO
Up to the second digit of the display time of P and the I / P displayed immediately before
-VOP (I / P-VO of interest among the VOPs that make up the GOV to be processed)
(I / P-VOP displayed immediately before P) (Last display I / P-VO
The difference value with the second digit of the display time of P) is obtained, the difference value is set in the variable D, and the process proceeds to step S5.

At step S5, whether or not the variable D is equal to 0, that is, the difference between time_code and the second digit of the display time of the attention I / P-VOP or the display time of the attention I / P-VOP is determined. It is determined whether the difference value between the second digit and the second digit of the display time of the I / P-VOP displayed immediately before that is 0 second. When it is determined in step S5 that the variable D is not equal to 0, that is, when the variable D is 1 or more, the process proceeds to step S6, and the MSB of modulo_time_base.
1 is added as (Most Significant Bit). That is, in this case, when modulo_time_base is, for example, 0B immediately after reset, it is set to 10B, and modu_time_base is modu_time_base.
When lo_time_base is, for example, 10B, 11
It is set to 0B.

Then, the process proceeds to step S7, where the variable D is 1
Is decremented, and the process returns to step S5. afterwards,
The processes of steps S5 to S7 are repeated until it is determined in step S5 that the variable D is equal to 0.
That is, modulo_time_base is changed to time_code
And the difference between the display time of the target I / P-VOP and the second digit of the display time, or up to the second digit of the display time of the target I / P-VOP and the I / P-VOP displayed immediately before that. "1" continues for the same number of seconds as the number of seconds corresponding to the difference value up to the second digit of the time, and 0 is added to the end.

If it is determined in step S5 that the variable D is equal to 0, the process proceeds to step S8 and VO
The P_time_increment is set to a time having a precision finer than the second precision of the display time of the target I / P-VOP, that is, a time in milliseconds, and the process ends.

[0269] Focused I / P-VOP obtained as described above
Modulo_time_base and VOP_time_increment of VL
In the C circuit 36, it is added to the I / P-VOP of interest and thereby included in the encoded bitstream.

Note that modulo_time_base and VOP_time_i
The ncrement and time_code are variable length coded in the VLC circuit 36.

Next, every time the VLC unit 36 receives a B-VOP forming a GOV to be processed, the B-VOP is focused on by the B-VOP.
As shown in the flow chart of FIG.
P modulo_time_base and VOP_time_increment are obtained and encoded.

That is, in the VLC device 36, first, in step S11, as in the case of step S1 of FIG. 29, modulo_time_base and VOP_time_incre.
ment is reset.

Then, the process proceeds to step S12, and the attention B-VO
P is the first I-VOP (Fir (Fir)
st I-VOP) is determined before it is displayed. In step S12, the attention B-VOP is
If it is determined that the GOV to be processed is displayed before the I-VOP displayed first, step S1.
Proceed to step 4, and display the time_code of the processing target GOV and the display time of the B-VOP of interest (here, the B-VOP displayed prior to the I-VOP displayed first in the processing target GOV). Is calculated and set in the variable D, and the process proceeds to step S15. Therefore, here, the variable D is set to the millisecond precision time (time to the millisecond digit) (in contrast, the variable D in FIG. 29 is set to the second precision as described above. Time is set).

Also, in step S12, the B-VO of interest is
If it is determined that P is displayed after the I-VOP displayed first in the GOVs to be processed, the process proceeds to step S14, and the display time of the B-VOP of interest and immediately before that. I / P-VOP displayed (I / P-VOP displayed immediately before the attention B-VOP among the VOPs constituting the GOV to be processed) (Last dis
play I / P-VOP) display time difference is obtained, the difference value is set in variable D, and the process proceeds to step S15.

In step S15, it is displayed whether or not the variable D is greater than 1, that is, the difference value between time_code and the display time of the B-VOP of interest, or the display time of the B-VOP of interest, and immediately before that. It is determined whether the difference value from the display time of the I / P-VOP is greater than 1 second. If it is determined in step S5 that the variable D is greater than 1, that is, 1 is added as the MSB of modulo_time_base,
It proceeds to step S17. In step S17, the variable D is decremented by 1, and the process returns to step S15. Then, the processes of steps S15 to S17 are repeated until it is determined in step S15 that the variable D is not greater than 1. I.e. this allows modulo_time_base
Is the difference between time_code and the display time of the B-VOP of interest, or the display time of the B-VOP of interest and the I / P displayed immediately before that.
-“1” continues for the same number of seconds as the time corresponding to the difference value from the display time of VOP, and 0 is added to the end.

Then, in step S15, the variable D
If it is determined that is not greater than 1, the process proceeds to step S8, and the value of the variable D at that time, that is, the time_code and the attention B
-The difference between the display time of the VOP or the display time of the B-VOP of interest and the difference value between the display time of the I / P-VOP displayed immediately before it and the position below the second digit (in milliseconds). Time) but VOP_
It is set to time_increment and the process ends.

[0277] m of the attention B-VOP obtained as described above
odulo_time_base and VOP_time_increment are VLC
In circuit 36, it is added to the B-VOP of interest and is thereby included in the encoded bitstream.

Next, in the IVLC unit 102, as described above, the VLC unit 36 divides the VOP sequence into GOVs and processes the divided VOP sequence to display the VOP display time in the encoded stream. , Each time the encoded data for each VOP is received, the VOP is recognized as a target VOP by processing, and the variable length decoding is performed so that the VOP is displayed at the display time. . That is, I
When the VLC device 102 receives the GOV, the time_c of the GOV is received.
Each time the ode is recognized and the I / P-VOP forming the GOV is received, the I / P-VOP is set as the target I / P-VOP, and the target I / P-VOP modulo_time_bas
The display time is calculated based on e and VOP_time_increment.

That is, in the IVLC device 102, first, in step S21, the focused I / P-VOP is the first I-VOP (First I-VOP) displayed in the GOVs to be processed.
Is determined. When it is determined in step S21 that the target I / P-VOP is the first I-VOP displayed in the processing target GOV, the process proceeds to step S23, and the variable T contains the time_code of the processing target GOV. After being set, the process proceeds to step S24.

In step S21, the target I / P-
If it is determined that the VOP is not the first I-VOP displayed in the GOVs to be processed, the process proceeds to step S22, and the I / O
I / P-VOP displayed immediately before the P-VOP (I / P-VOP displayed immediately before the focused I / P-VOP among the VOPs forming the GOV to be processed)
VOP) (Last display I / P-VOP) up to the second digit of the display time is set in the variable T, and the process proceeds to step S24.

In step S24, it is determined whether modulo_time_base added to the target I / P-VOP is equal to 0B. In step S24, when it is determined that modulo_time_base added to the attention I / P-VOP is not equal to 0B, that is, m added to the attention I / P-VOP.
If odulo_time_base includes 1, step S25
Then, the MSB 1 of modulo_time_base is deleted and the process proceeds to step S26. In step S26, the variable T is incremented by 1 second, and the process returns to step S24, and thereafter, in step S24, steps S24 to S26 are repeated until modulo_time_base added to the attention I / P-VOP is equal to 0B. The process of is repeated. As a result, the variable T is incremented by the number of seconds corresponding to the number 1 of modulo_time_base that was initially added to the target I / P-VOP.

Then, in step S24, the attention I /
When it is determined that modulo_time_base added to the P-VOP is equal to 0B, the process proceeds to step S27 and the variable T
Is added to the time of millisecond precision represented by VOP_time_increment, the added value is recognized as the display time of the I / P-VOP of interest, and the process ends.

Next, when the IVLC unit 102 receives the B-VOP forming the GOV, it follows the modulo_time_base and VOP of the B-VOP of interest according to the flowchart of FIG.
The display time is calculated based on _time_increment.

That is, in the IVLC device 102, first, in step S31, the focused B-VOP is the processing target G.
It is determined whether the OV is displayed before the I-VOP displayed first (First I-VOP). When it is determined in step S31 that the focused B-VOP is displayed before the I-VOP displayed first in the GOVs to be processed, the process proceeds to step S33, and then step S33. In steps S37 to S37, the same processing as that in steps S23 to S27 of FIG.

On the other hand, in step S31, the target B-VO
When it is determined that P is displayed after the I-VOP displayed first in the GOVs to be processed, the process proceeds to step S32, and then steps S32, S34 to S34.
37, steps S22, S24 to S of FIG.
The display time of the focused B-VOP is obtained by performing the same processing as that in 27.

Next, in the second method, the time between the display time of the I-VOP and the predicted display time of the B-VOP is calculated up to the order of seconds, and the value is expressed by modulo_time_base. Then, the millisecond precision of the B-VOP display time is set to VOP_time_incr
Express in ement. That is, in VM6.0, as described above,
The weight for I-VOP or P-VOP used as a reference image at the time of predictive coding of B-VOP, from B-VOP, I- sandwiching it
To decide based on the temporal distance to VOP or P-VOP, the temporal distance is set to VOP_time_incr for B-VOP.
However, the VOP_time_increment for I-VOP and P-VOP is different from the time from the sync point indicated by the immediately encoded / decoded modulo_time_base. If the display time and the display time of the I-VOP or P-VOP sandwiching the display time are known, the temporal distance between them can be obtained by taking the difference, and therefore, only the VOP_time_increment for the B-VOP can be obtained. , V-time_increme for I-VOP and P-VOP
There is little need to treat it differently from nt. Rather, from the viewpoint of processing efficiency, VOP_t of all VOPs of I, B, P
ime_incrment (detailed time information), and modulo_tim
It is desirable to handle e_base (second precision time information) in the same way.

Therefore, in the second method, modulo_time_base and VOP_time_increment for B-VOP are set to I / P.
-The handling is the same as for VOP.

For example, in the case shown in FIG. 27, according to the second method, modulo_time_base and VOP_ti.
time_ about GOV when encoding me_increment
FIG. 33 shows the relationship between code and modulo_time_base and VOP_time_increment.

That is, also in the second method, modulo_time_base
Is allowed not only for I-VOP and P-VOP but also for B-VOP. And modulo_time added to B-VOP
Similarly to modulo_time_base added to I / P-VOP, _base also represents switching of synchronization points.

Further, in the second method, the value obtained by subtracting the time of the sync point indicated by modulo_time_base added to the B-VOP from the display time of the B-VOP is the VOP.
It is set as _time_increment.

Therefore, according to the second method, in FIG. 27, I1 displayed between the first sync point of the GOV, which is the time represented by the time_code of the GOV, and the sync point indicated by the time time_code + 1 second, is displayed. Or modulo_time_base of B2
Is set to 0B, and each VOP_ti
The me_increment is set to a value in milliseconds that is lower than the second place of the display time of I1 or B2. Also, the time
From the sync point indicated by time_code + 1 second, time time_code
The modulo_time_base of B3 or B4 displayed up to the synchronization point indicated by +2 seconds is 10B, and the VOP_time_increment of each is 3 mm below the second of the display time of B3 or B4. The value in seconds is set. Furthermore, the modulo_time_base of P5 displayed from the sync point indicated by time time_code + 2 seconds to the sync point indicated by time time_code + 3 seconds is 11
It is set to 0B, and its VOP_time_increment is P
A value in milliseconds below the second place of the display time of 5 is set.

In FIG. 27, for example, assuming that the display time of I1 is 0h: 12m: 35s: 350ms and the display time of B4 is 0h: 12m: 36s: 550ms, as described above, I
Modulo_time_base of 1 or B4 is, as described above,
It is set to 0B or 10B, respectively. The VOP_time_increment of I1 or B4 is 350 ms or 550 ms, which is the millisecond unit of the display time, respectively.

Modulo_t by the second method as described above
As for the processing for ime_base and VOP_time_increment, for example, as in the case of the first method, FIG. 9 and FIG.
15 and the IVLC device 102 shown in FIGS. 15 and 16.

That is, in the VLC device 36, I / P-VOP is modulo_time_in the same manner as in FIG.
base and VOP_time_increment are required.

Regarding the B-VOP, each time the VLC unit 36 receives a B-VOP forming a GOV, that B-VOP is received.
According to the flowchart of FIG. 34, assuming that the VOP is the B-VOP of interest, the modulo_time_base and VOP_time_in of the B-VOP of interest
crement is required.

That is, in the VLC device 36, first, in step S41, as in the case of step S1 of FIG. 29, modulo_time_base and VOP_time_i.
ncrement is reset.

[0297] Then, the processing proceeds to step S42, and the attention B-VO
Among the GOVs that P is processing (GOVs to be processed),
It is determined whether or not it is displayed before the first I-VOP (First I-VOP) displayed. If it is determined in step S12 that the B-VOP of interest is displayed prior to the I-VOP displayed first in the GOVs to be processed, the process proceeds to step S44, and the GOVs to be processed are processed. Time_code and the second precision of the display time of the B-VOP of interest, that is, the difference between the time_code and the second digit of the display time of the B-VOP of interest is obtained and set to the variable D,
It proceeds to step S45.

In step S42, the target B-VO
When it is determined that P is displayed after the I-VOP displayed first in the GOVs to be processed, the process proceeds to step S43, and up to the second digit of the display time of the attention B-VOP. And the I / P-VOP displayed immediately before that (the I / P-VO displayed immediately before the target B-VOP among the VOPs forming the processing target GOV)
P) (Last display I / P-VOP), the difference value up to the second digit of the display time is obtained, the difference value is set in the variable D, and the process proceeds to step S45.

In step S45, it is determined whether or not the variable D is equal to 0, that is, the difference between time_code and the second digit of the display time of the attention B-VOP, or the second digit of the display time of the attention B-VOP. Then, it is determined whether or not the difference value between the display time of the I / P-VOP displayed immediately before that and the second digit is 0 second. When it is determined in step S45 that the variable D is not equal to 0, that is, when the variable D is 1 or more, the process proceeds to step S46, and the MSB of modulo_time_base.
Is added as 1.

Then, the process proceeds to step S47, the variable D is decremented by 1, and the process returns to step S45. After that, the processes of steps S45 to S47 are repeated until it is determined in step S45 that the variable D is equal to 0. That is, as a result, modulo_time_base is ti
The difference between me_code and the second digit of the display time of the B-VOP of interest, or the second digit of the display time of the B-VOP of interest and the second of the display time of the I / P-VOP displayed immediately before it. "1" continues for the same number of seconds corresponding to the difference value up to the digit
The value with 0 added to the end is set.

Then, in step S45, the variable D
When it is determined that is equal to 0, the process proceeds to step S48, the VOP_time_increment is set to a time having a precision finer than the second precision of the display time of the B-VOP of interest, that is, a time in milliseconds, and the process ends.

On the other hand, in the IVLC unit 102, the I / P-VOP is set in the same manner as in the case of FIG.
The display time is calculated based on dulo_time_base and VOP_time_increment.

For B-VOP, IVLC unit 1
In 02, every time the B-VOP that constitutes the GOV is received,
With the B-VOP as the attention B-VOP, according to the flowchart in FIG. 35, the modulo_time_base and VOP_t of the attention B-VOP are set.
The display time is calculated based on ime_increment.

That is, in the IVLC device 102, first, in step S51, the B-VOP of interest is the processing target G.
It is determined whether or not the OV is displayed before the I-VOP displayed first (First I-VOP). In step S51, the attention B-VOP is the processing target G.
When it is determined that the OV is displayed prior to the I-VOP displayed first, the process proceeds to step S52, the time_code of the processing target GOV is set in the variable T, and the step S54 is performed. Proceed to.

In step S51, the B-VO of interest is noted.
When it is determined that P is displayed after the I-VOP displayed first in the GOV to be processed, the process proceeds to step S53, and the I / I displayed immediately before the B-VOP of interest is displayed. P-VOP
(Of the VOPs that compose the GOV to be processed, the I / P-VOP displayed immediately before the B-VOP of interest) (Last display I / P-VOP) is displayed in the variable T up to the second digit. After being set, the process proceeds to step S54.

[0306] In step S54, it is determined whether modulo_time_base added to the B-VOP of interest is equal to 0B. When it is determined in step S54 that modulo_time_base added to the attention B-VOP is not equal to 0B, that is, modulo_time added to the attention B-VOP.
When the time_base includes 1, the process proceeds to step S55, the MSB 1 of modulo_time_base is deleted, and the process proceeds to step S56. In step S56, the variable T is 1
It is incremented by seconds and returns to step S54, and thereafter, the processes of steps S54 to S56 are repeated until it is determined in step S54 that modulo_time_base added to the attention B-VOP is equal to 0B. As a result, the variable T is incremented by the number of seconds corresponding to the number 1 of modulo_time_base that was initially added to the target B-VOP.

[0307] Then, in step S54, the attention B-
When it is determined that modulo_time_base added to VOP is equal to 0B, the process proceeds to step S57 and the variable T is set to
The time with millisecond precision represented by VOP_time_increment is added, the added value is recognized as the display time of the B-VOP of interest, and the process ends.

As described above, the coding start absolute time is coded in the structure (layer) of the coded bit stream.
The GOV layer is introduced so that this GOV layer can be inserted not only at the beginning of the bitstream, but also at an appropriate position, and the modulo_time_base and V
Since the definition of OP_time_increment has been changed as described above, the display time (absolute time) of each VOP can be obtained in all cases, regardless of the arrangement of VOP picture types and the time interval between adjacent VOPs. It will be possible.

Therefore, in the encoder, the absolute start time of encoding is encoded in GOV units and
By encoding dulo_time_base and VOP_time_increment and including them in the encoded bitstream, the decoder decodes the encoding start absolute time in GOV units and
Modulo_time_base and VOP_time_increment can be decoded, and the display time of each VOP can be decoded from them, it is possible to efficiently perform random access in GOV units.

Note that if the number of 1's added to modulo_time_base is simply increased according to the switching of the sync points, one hour (3600 seconds) after the time indicated by time_code (however, GOV , But with a VOP equivalent to that much time), modulo_time_b
Since ase is composed of 1 in 3600 bits and 0 in 1 bit, the number of bits becomes 3601 bits.

Therefore, in MPEG4, modulo_time_base
Are defined to be reset at the first I / P-VOP that appears after the switching of sync points.

Therefore, for example, as shown in FIG.
V is the time represented by the time_code, I1 and B2 displayed between the first sync point of GOV and the sync point indicated by time time_code + 1 seconds, time time_code + 1
B3 and B4 displayed between the sync point indicated by seconds and the sync point indicated by time_code + 2 seconds, the time
From the sync point indicated by time_code + 2 seconds, the time time_code
From the sync point indicated by P5 and B6, time_code + 3 seconds, which is displayed until the sync point indicated by +3 seconds,
When it is composed of B7 displayed between the sync point indicated by time time_code + 4 seconds and B8 displayed between the sync point indicated by time time_code + 4 seconds and the sync point indicated by time time_code + 5 seconds Is the modulo_time_ of I1 and B2 displayed between the first sync point of GOV and the sync point indicated by time time_code + 1 second.
The base is set to 0B.

Also, the modulo_time_base of B3 and B4 displayed from the synchronization point indicated by time time_code + 1 second to the synchronization point indicated by time time_code + 2 seconds.
Is 10B. Furthermore, the modulo_time_base of P5 displayed between the sync point indicated by time time_code + 2 seconds and the sync point indicated by time time_code + 3 seconds.
Is 110B.

Since P5 is the first P-VOP displayed after switching from the first sync point of GOV to the sync point indicated by time_code + 1 second, modulo_
The time_base is reset to 0B, and the modulo_time_base of B6 displayed thereafter is the synchronization point referred to when the display time of P5 is obtained, that is, the time time in this case.
It is set by regarding the sync point indicated by _code + 2 seconds as the first sync point of GOV. Therefore, B6 modulo_time_base
Is set to 0B.

After that, the modulo_time_base of B7 displayed between the synchronization point indicated by time time_code + 3 seconds and the synchronization point indicated by time time_code + 4 seconds is 10B.
From the synchronization point indicated by time_code + 4 seconds,
The modulo_time_base of B8 displayed up to the synchronization point indicated by time time_code + 5 seconds is 110B.

In the processing on the encoder side (VLC unit 36) described with reference to FIGS. 29, 30, and 34, modulo_time_base is set as described above.

In this case, on the decoder side (IVLC unit 102), when the I / P-VOP displayed first after the switching of the synchronization point is detected, the mod added to it is detected.
It is necessary to cumulatively add the number of seconds indicated by ulo_time_base to the time_code to obtain the display time. That is, for example, in the case shown in FIG. 36, the display times of I1 to P5 are added to the time_code for each VOP. Added to modu
It can be calculated by adding the number of seconds corresponding to lo_time_base and VOP_time_increment, but the display time of B6 to B8 displayed after P5 that is first displayed after the switching of the sync point is added to each VOP in time_code. Being modulo
In addition to adding the number of seconds corresponding to _time_base and VOP_time_increment, it is necessary to add 2 seconds corresponding to modulo_time_base of P5 to obtain the value. In FIG. 31, FIG. 32, and FIG. The described process is thus performed to obtain the display time.

Next, the encoder and decoder described above can be realized by dedicated hardware, or can also be realized by causing a computer to execute a program for causing the above-described processing. You can

FIG. 37 is a block diagram of the encoder of FIG.
2 illustrates a configuration example of an embodiment of a computer that functions as a decoder of the.

A ROM (Read Only Memory) 201 stores, for example, a boot program and the like. CPU
(Central Processing Unit) 202 is, for example, HD
The program stored in (HardDisk) 206 is stored in RAM
Various processes are performed by expanding the program on the (Read Only Memory) 203 and executing it. RAM20
3 is a program executed by the CPU 202 and the CPU 2
The data necessary for the process 02 is temporarily stored. The input unit 204 includes, for example, a keyboard and a mouse, and is operated when inputting necessary commands and data. The output unit 205 includes, for example, a display, and displays data according to the control of the CPU 202. The HD 206 stores a program to be executed by the CPU 202, image data to be encoded, encoded data (encoded bit stream), image data after decoding, and the like. The communication I / F (Interface) 207 controls the communication with the outside to receive, for example, the image data to be encoded from the outside, or to transmit the encoded bit stream after encoding to the outside. It is designed to do. The communication I / F 207 is also configured to receive an encoded bit stream encoded externally, and also transmit image data after decoding to the outside.

Computer C configured as described above
By causing the PU 202 to execute the program for performing the above-described processing, this computer functions as the encoder shown in FIG. 1 or the decoder shown in FIG.

In this embodiment, VOP_time_incre
The ment represents the display time of the VOP in units of 1 ms, but the VOP_time_increment may be, for example, as follows. That is, the value from one sync point to the next sync point is divided into N, and a value indicating the number of the division point from the one sync point is the division point corresponding to the display time of the VOP. , VOP_time_increment. When VOP_time_increment is defined in this way, if N = 1000, VOP_time_increment becomes
The VOP display time is expressed in 1 ms units. In this case, the decoder needs information about how many divisions are made from one synchronization point to the next synchronization point. However, the number of divisions between the synchronization points is set in advance. Good, or include it in the hierarchy above the GOV layer,
It may be provided to the decoder.

[0323]

According to the image coding method of the present invention, a plurality of VOPs are grouped and absolute time information indicating the absolute time when the coding of the VOPs of each group is started is added in group units. Further, second precision time information representing relative time in the group with second precision is generated, and from the second precision time information immediately before the display time of each I-VOP, P-VOP, or B-VOP, the respective precision time information is displayed. Detailed time information is generated that represents the time until the display time with a precision finer than the second precision. And I-VOP, P-V
As information indicating the display time of OP or B-VOP,
Second precision time information and detailed time information correspond to IV
It is added to OP, P-VOP, or B-VOP, respectively. In this case, as the second precision time information about the predetermined VOP, the time from the absolute time information to the display time of the predetermined VOP is represented by the second precision or the predetermined VOP.
I-VOP or P-VOP displayed immediately before OP
The time from the display time of to the display time of the predetermined VOP is represented with second precision, and the absolute time information is
For each I-VOP, P-VOP, or B-VOP
Adds the second precision time information and detailed time information
The calculated time is I-VOP, P-VOP, or B-V
The display time of each OP is set. Therefore, it is possible to perform random access in units of groups with respect to the encoded result, and it is possible to prevent the second precision time information from having an enormous number of bits, for example.

According to the image decoding method and the image decoding apparatus of the present invention, the absolute time information includes I-VOP, P-VOP,
Or the second precision time added to each B-VOP
By adding information and detailed time information, I-VO
Display time of P, P-VOP, or B-VOP
Is required, I-VOP, P-VOP, or B-VO
P is decoded according to the corresponding display time. In this case, as the second precision time information for the predetermined VOP, the time from the absolute time information to the display time of the predetermined VOP is expressed in second precision, or the I-displayed immediately before the predetermined VOP. The time from the display time of the VOP or the P-VOP to the display time of the predetermined VOP, which is expressed in seconds, is used. Therefore, it is possible to perform random access in units of groups to the encoded bitstream and perform decoding. Also, for example,
It is possible to prevent the second precision time information from having an enormous number of bits.

[0325]

[0326]

[0327]

[0328]

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration example of an embodiment of an encoder to which the present invention has been applied.

FIG. 2 is a diagram for explaining that the position and size of a VO change with time.

FIG. 3 is a block diagram showing a configuration example of VOP encoding sections 3 _{1 to} 3 _N in FIG.

FIG. 4 is a diagram for explaining spatial scalability.

FIG. 5 is a diagram for explaining spatial scalability.

FIG. 6 is a diagram for explaining spatial scalability.

FIG. 7 is a diagram for explaining spatial scalability.

FIG. 8 is a diagram for explaining a method of determining VOP size data and offset data.

9 is a block diagram showing a configuration example of a lower layer encoding unit 25 in FIG.

10 is a block diagram showing a configuration example of an upper layer encoding unit 23 in FIG.

FIG. 11 is a diagram for explaining spatial scalability.

FIG. 12 is a diagram for explaining time scalability.

FIG. 13 is a block diagram showing a configuration example of an embodiment of a decoder to which the present invention has been applied.

14 is a block diagram showing another configuration example of the VOP decoding units 72 _{1 to} 72 _N of FIG.

15 is a block diagram showing a configuration example of a lower layer decoding unit 95 in FIG.

16 is a block diagram illustrating a configuration example of an upper layer decoding unit 93 in FIG.

[Fig. 17] Fig. 17 is a diagram illustrating the syntax of a bitstream obtained by scalable coding.

FIG. 18 is a diagram showing the syntax of VS.

[Fig. 19] Fig. 19 is a diagram illustrating the syntax of a VO.

[Fig. 20] Fig. 20 is a diagram illustrating the syntax of a VOL.

FIG. 21 is a diagram illustrating the syntax of VOP.

[Fig. 22] Fig. 22 is a diagram illustrating the relationship between modulo_time_base and VOP_time_increment.

FIG. 23 is a diagram showing the syntax of a bitstream according to the present invention.

FIG. 24 is a diagram showing the syntax of GOV.

FIG. 25 is a time_code of the GOV layer, and modulo_time_base and V of the I-VOP at the head of the GOV.
It is a figure which shows the encoding method of OP_time_increment.

FIG. 26 is a time_code of a GOV layer and a modulo_t of a B-VOP located before the I-VOP at the head of the GOV.
It is a figure which shows the encoding method of ime_base and VOP_time_increment.

[Fig. 27] Fig. 27 is a diagram illustrating the relationship between modulo_time_base and VOP_time_increment when the definitions are not changed.

FIG. 28: B-VOP modulo_time_base and VOP_time_i
It is a figure which shows the encoding process by the 1st method of ncrement.

FIG. 29: modulo_time_base and VOP_ti of I / P-VOP
It is a flowchart which shows the encoding process by the 1st and 2nd method of me_increment.

FIG. 30: B-VOP modulo_time_base and VOP_time_i
It is a flowchart which shows the encoding process by the 1st method of ncrement.

FIG. 31 is an I / O coded by the first and second methods.
It is a flowchart which shows the decoding process of modulo_time_base and VOP_time_increment of P-VOP.

FIG. 32: mo of B-VOP coded by the first method
It is a flowchart which shows the decoding process of dulo_time_base and VOP_time_increment.

[Fig. 33] B-VOP modulo_time_base and VOP_time_i
It is a figure which shows the encoding process by the 2nd method of ncrement.

FIG. 34 is a B-VOP modulo_time_base and VOP_time_i
It is a flowchart which shows the encoding process by the 2nd method of ncrement.

FIG. 35: mo of B-VOP encoded by the second method
It is a flowchart which shows the decoding process of dulo_time_base and VOP_time_increment.

[Fig. 36] Fig. 36 is a diagram for describing modulo_time_base.

[Fig. 37] Fig. 37 is a block diagram illustrating a configuration example of another embodiment of an encoder and a decoder to which the present invention has been applied.

FIG. 38 is a block diagram showing a configuration of an example of a conventional encoder.

FIG. 39 is a block diagram showing a configuration of an example of a conventional decoder.

[Explanation of symbols]

1 VO component, 2 _{1 to} 2 _N VOP component, 3 ₁
To 3 _N VOP encoder, 4 multiplexer, 21
Image layering unit, 23 upper layer coding unit, 24 resolution conversion unit, 25 lower layer coding unit, 26 multiplexing unit, 31 frame memory, 32 motion vector detector, 33 arithmetic unit, 34 DCT unit, 35
Quantizer, 36 VLC device, 38 inverse quantizer,
39 IDCT device, 40 arithmetic unit, 41 frame memory, 42 motion compensator, 53 frame memory,
71 demultiplexing unit, 72 _{1 to} 72 _N VOP decoding unit,
73 image reconstruction unit, 91 demultiplexing unit, 93
Upper layer decoding unit, 94 resolution conversion unit, 95 lower layer decoding unit, 102 IVLC unit, 103 inverse quantizer, 104 IDCT unit, 105 arithmetic unit, 1
06 frame memory, 107 motion compensator, 11
2 frame memory, 201 ROM, 202 CP
U, 203 RAM, 204 input section, 205 output section, 206 HD, 207 communication I / F

─────────────────────────────────────────────────── ─── Continuation of the front page (56) References Japanese Unexamined Patent Publication No. 5-128823 (JP, A) Special Table 11-513222 (JP, A) International Publication 95/23411 (WO, A1) NPEG-4 latest information, IEICE Technical Report, March 19, 1997, IE96-141, p. 1-8 (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N ^7/ 24-7/68

Claims (3)

(57) [Claims] 1. An image encoding method for encoding an image for each VOP (Video Object Plane) which is an object constituting the image, and outputting an encoded bit stream obtained as a result, which is intra-encoded. I-VOP (Intra-VO
P) and a VOP coded by either intra coding or forward predictive coding as P-VOP (Predicti
ve-VOP) and a VOP coded by any of intra coding, forward predictive coding, backward predictive coding, or bidirectional predictive coding, as a B-VOP (Biderectionally Pr
edictive-VOP), each of the plurality of VOPs is grouped, and the VOP of each group is
A first adding step of adding absolute time information representing the absolute time when the encoding of the above is started to the group unit, and second precision time information for generating second precision time information representing relative time within the group with second precision Detailed time information representing the generation step and the time from the second precision time information immediately before the display time of each of the I-VOP, P-VOP, and B-VOP to each display time with a precision finer than the second precision. And a detailed time information generating step of generating the detailed time information, and the second precision time information and the detailed time information as the information indicating the display time of the I-VOP, P-VOP, or B-VOP. -VOP or B-VOP, and a second addition step of adding each to the predetermined VO in the second precision time information generation step.
As the second precision time information about P, the time from the absolute time information to the display time of the predetermined VOP,
Generated in seconds precision or the time from the display time of the I-VOP or P-VOP displayed immediately before the predetermined VOP to the display time of the predetermined VOP in seconds precision and, to the absolute time information, the I-VOP, P-VOP, or
Or the second precision added to each B-VOP
The time obtained by adding the time information and the detailed time information to the I-
Display of VOP, P-VOP, or B-VOP
An image coding method characterized by setting the time . 2. An image decoding method for decoding an encoded bit stream obtained by encoding an image for each VOP (Video Object Plane) which is an object constituting the image, wherein an intra-encoded VOP is I-VOP (Intra-VO
P) and a VOP coded by either intra coding or forward predictive coding as P-VOP (Predicti
ve-VOP) and a VOP coded by any of intra coding, forward predictive coding, backward predictive coding, or bidirectional predictive coding, as a B-VOP (Biderectionally Pr
edictive-VOP) and each of the plurality of VOPs are grouped, and the VO of each group is
Absolute time information indicating the absolute time at which the encoding of P is started is added to the group unit, and the relative time within the group is represented by the second precision time information by the second precision and the I-VOP, P-VOP. , Or B
-Detailed time information indicating the time from the second precision time information immediately before each display time of each VOP to each display time with a precision finer than the second precision, as the information indicating the display time, the corresponding I -VOP, P-VOP,
Alternatively, when added to each B-VOP, the absolute time information includes the I-VOP, P-VOP, or
Or the second precision added to each B-VOP
By adding the time information and the detailed time information, the I-
Display of VOP, P-VOP, or B-VOP
A display time calculation step of calculating the time, the I-VOP, a P-VOP or B-VOP,, and a decoding step of decoding according to the corresponding display time, as the second precision time information for a given VOP, The time from the absolute time information to the display time of the predetermined VOP, which is expressed in seconds, or the predetermined VO.
An image decoding method characterized in that the time from the display time of the I-VOP or P-VOP displayed immediately before P to the display time of the predetermined VOP is expressed with second precision. . 3. An image is an object that composes the image
Encoding for each VOP (Video Object Plane)
Image decoding that decodes the encoded bitstream obtained by
A device, An intra-coded VOP is converted into an I-VOP (Intra-VO
P) and intra or forward predictive coding
VOP coded by either P-VOP (Predicti
ve-VOP), intra coding, forward prediction coding, backward prediction
Either in the measurement coding or in the bidirectional predictive coding
The coded VOP is a B-VOP (Biderectionally Pr
edictive-VOP) A plurality of the VOPs are grouped, and the VO of each group is
Absolute time information indicating the absolute time when the encoding of P is started is
It is added to each group, The relative time within the group is shown with the second precision.
Degree time information and the I-VOP, P-VOP, or B
-Second precision time information immediately before the display time of each VOP
From the information to the respective display time, the second accuracy
Detailed time information expressed with fine accuracy indicates the display time.
As the information, the corresponding I-VOP, P-VOP,
Or when added to B-VOP respectively, The absolute time information includes the I-VOP, P-VOP,
Or the second precision added to each B-VOP
By adding the time information and the detailed time information, the I-
Display of VOP, P-VOP, or B-VOP
A display time calculating means for obtaining the time, The I-VOP, P-VOP, or B-VOP is paired with
Decoding means for decoding according to the corresponding display time Equipped
e, As the second precision time information for a predetermined VOP,
From the absolute time information to the display time of the predetermined VOP
Of time in seconds, or the predetermined VO
I-VOP or P-VOP displayed immediately before P
Time from the display time to the display time of the predetermined VOP
Is expressed in seconds accuracy is used. Characterized by
Image decoding device.