JP3345887B2 - Image encoding device and image encoding method, image decoding device and image decoding method, and recording medium - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to an image encoding device and an image encoding method, an image decoding device and an image decoding method, and a recording medium. In particular, for example, moving image data is recorded on a recording medium such as a magneto-optical disk or a magnetic tape, and is reproduced and displayed on a display or the like, a video conference system, a video telephone system, a broadcasting device, a multimedia database. Used for transmitting moving image data from the transmitting side to the receiving side via a transmission path, such as a search system, and for displaying and editing or recording the received moving image data on the receiving side. TECHNICAL FIELD The present invention relates to an image encoding device and an image encoding method, an image decoding device and an image decoding method, and a recording medium that are suitable.

[0002]

2. Description of the Related Art For example, in a system for transmitting moving image data to a remote place, such as a video conference system or a video telephone system, image data is converted into a line correlation or a frame in order to use a transmission path efficiently. The compression encoding is performed using the inter-correlation.

A typical high-efficiency video coding scheme is Moving Picture Experts Group (MPEG).
(Moving picture coding for storage). This is ISO-I
Discussed in EC / JTC1 / SC2 / WG11,
It has been proposed as a standard, and employs a hybrid method combining motion compensation prediction coding and DCT (Discrete Cosine Transform) coding.

[0004] In MPEG, several profiles and levels are defined in order to support various applications and functions. The most basic is the main profile main level (MP @ ML (Main Profile
at Main Level)).

FIG. 53 is a diagram showing MP @ M in the MPEG system.
5 shows an exemplary configuration of an L encoder.

[0006] Image data to be encoded is input to a frame memory 31 and is temporarily stored. Then, the motion vector detector 32 reads out the image data stored in the frame memory 31 in units of macroblocks composed of, for example, 16 pixels × 16 pixels, and detects the motion vector.

Here, the motion vector detector 32 converts the image data of each frame into an I picture (intra-frame coding), a P picture (forward prediction coding), or a B picture.
It is processed as one of the pictures (bidirectional predictive coding). It should be noted that it is predetermined, for example, which of I, P, and B pictures to process the image of each sequentially input frame as a picture (for example, I, B, P, B, P,...). .. Processed as B and P).

That is, the motion vector detector 32 refers to a predetermined reference frame in the image data stored in the frame memory 31, and the reference frame and the current encoding target are By performing pattern matching (block matching) with a small block (macroblock) of 16 pixels × 16 lines of a frame that is present, a motion vector of the macroblock is detected.

[0009] Here, in MPEG, the image prediction modes include four modes: intra coding (intra-frame coding), forward prediction coding, backward prediction coding, and bidirectional prediction coding.
There are types, I-pictures are intra-coded, P-pictures are coded by either intra-coding or forward-prediction coding, and B-pictures are intra-coded, forward-prediction-coded, backward-prediction-coded, or both. Encoded by any of the method predictive encodings.

That is, the motion vector detector 32 sets the intra coding mode as the prediction mode for the I picture. In this case, the motion vector detector 32
The motion vector is not detected, and the prediction mode (intra prediction mode) is output to the VLC (variable length coding) unit 36 and the motion compensator 42.

The motion vector detector 32 performs forward prediction on a P picture and detects the motion vector. Further, the motion vector detector 32 compares a prediction error caused by performing forward prediction with, for example, a variance of a macroblock to be coded (a macroblock of a P picture). As a result of the comparison, when the variance of the macroblock is smaller than the prediction error, the motion vector detector 32 sets the intra coding mode as the prediction mode, and outputs it to the VLC unit 36 and the motion compensator 42. If the prediction error caused by performing the forward prediction is smaller, the motion vector detector 32 sets the forward prediction encoding mode as the prediction mode, and sets the VLC unit 36 and the motion compensator 42 together with the detected motion vector.
Output to

Further, the motion vector detector 32 performs forward prediction, backward prediction, and bidirectional prediction on the B picture, and detects respective motion vectors. Then, the motion vector detector 32 performs forward prediction, backward prediction,
And a minimum prediction error of the bidirectional prediction (hereinafter, appropriately referred to as a minimum prediction error), and the minimum prediction error and the variance of the encoding target macroblock (the macroblock of the B picture), for example. Compare with As a result of the comparison, if the variance of the macroblock is smaller than the minimum prediction error, the motion vector detector 32 sets the intra coding mode as the prediction mode, and sets the VLC unit 3
6 and the motion compensator 42. If the minimum prediction error is smaller, the motion vector detector 32 sets the prediction mode in which the minimum prediction error is obtained as the prediction mode, and sets the VLC unit 3 together with the corresponding motion vector.
6 and the motion compensator 42.

The motion compensator 42 includes a motion vector detector 3
When both the prediction mode and the motion vector are received from 2,
According to the prediction mode and the motion vector, the encoded and already locally decoded image data stored in the frame memory 41 is read, and the read image data is used as the prediction image data by the calculator 33.
And 40.

The arithmetic unit 33 reads from the frame memory 31 the same macroblock as the image data read from the frame memory 31 by the motion vector detector 32, and calculates the difference between the macroblock and the predicted image from the motion compensator 42. Is calculated. This difference value is supplied to the DCT unit 34.

On the other hand, when only the prediction mode is received from the motion vector detector 32, that is, when the prediction mode is the intra-coding mode, the motion compensator 42 does not output a predicted image. In this case, the operator 33 (the operator 40
Does not perform any particular processing, and converts the macroblock read from the frame memory 31 into the DCT unit 34 without any processing.
Output to

The DCT unit 34 performs a DCT process on the output data of the arithmetic unit 33, and obtains the resulting D
The CT coefficient is supplied to the quantizer 35. Quantizer 35
Then, the amount of data stored in the buffer 37 (the amount of data stored in the buffer 37) (buffer feedback)
, A quantization step (quantization scale) is set. In the quantization step, the DCT from the DCT unit 34 is set.
The coefficients are quantized. The quantized DCT coefficients (hereinafter, appropriately referred to as quantization coefficients) are supplied to the VLC unit 36 together with the set quantization steps.

In the VLC unit 36, the quantized coefficient supplied from the quantizer 35 is converted into a variable length code such as a Huffman code and output to a buffer 37. further,
The VLC unit 36 performs a quantization step from the quantizer 35,
The prediction mode (mode indicating which of intra coding (intra-picture predictive coding), forward predictive coding, backward predictive coding, and bidirectional predictive coding has been set) and motion from the motion vector detector 32 The vector is also variable-length coded, and the resulting coded bit stream is
Output to the buffer 37.

The buffer 37 temporarily stores the coded bit stream from the VLC unit 36 to smooth the data amount, and outputs the data amount to, for example, a transmission path or records it on a recording medium.

The buffer 37 outputs the data storage amount to the quantizer 35, and the quantizer 35 sets a quantization step according to the data storage amount from the buffer 37. That is, the quantizer 35 outputs the buffer 3
When 7 is about to overflow, the quantization step is increased, thereby reducing the data amount of the quantization coefficient. When the buffer 37 is about to underflow, the quantizer 35 reduces the quantization step, thereby increasing the data amount of the quantization coefficient. Thus, the overflow and the underflow of the buffer 37 are prevented.

The quantization coefficient and the quantization step output from the quantizer 35 are determined not only by the VLC unit 36 but also by the inverse quantizer 3.
8 as well. Inverse quantizer 38
Then, the quantized coefficient from the quantizer 35 is inversely quantized in accordance with a quantization step from the quantizer 35, and is thereby converted into a DCT coefficient. The DCT coefficient is supplied to an IDCT unit (inverse DCT unit) 39. I
In the DCT unit 39, the DCT coefficient is subjected to inverse DCT processing, and data obtained as a result of the processing is supplied to the arithmetic unit 40.

The arithmetic unit 40 is supplied with the same data as the predicted image supplied to the arithmetic unit 33 from the motion compensator 42, as described above, in addition to the output data of the IDCT unit 39. The arithmetic unit 40 locally decodes the original image data by adding the output data (prediction residual (difference data)) of the IDCT unit 39 and the predicted image data from the motion compensator 42, and locally decodes the original image data. The output image data (local decoded image data) is output (however, when the prediction mode is the intra coding, the output data of the IDCT unit 39 passes through the arithmetic unit 40 and is directly output to the locally decoded image data. The data is supplied to the frame memory 41 as data).
The decoded image data is the same as the decoded image data obtained on the receiving side.

The decoded image data (local decoded image data) obtained by the arithmetic unit 40 is supplied to and stored in the frame memory 41, and then inter-coded (forward predictive coding, backward predictive coding, quantitative prediction). It is used as reference image data (reference frame) for the image to be encoded.

Next, FIG. 54 shows an MPE for decoding an encoded bit stream output from the encoder of FIG.
5 shows an exemplary configuration of a decoder of MP @ ML in G.

An encoded bit stream transmitted via a transmission path is received by a receiving device (not shown), or an encoded bit stream recorded on a recording medium is reproduced by a reproducing device (not shown). Provided and stored.

The IVLC unit (inverse VLC unit (variable length decoder)) 102 reads out the coded bit stream stored in the buffer 101 and performs variable length decoding to convert the coded bit stream into macroblock units. , Motion vector, prediction mode, quantization step, and quantization coefficient. Among these data, the motion vector and the prediction mode are supplied to the motion compensator 107, and the quantization step and the quantization coefficient of the macroblock are supplied to the inverse quantizer 103.

The inverse quantizer 103 converts the quantization coefficient of the macro block supplied from the IVLC unit 102 into I
The inverse quantization is performed according to the quantization step supplied from the VLC unit 102, and the resulting DCT coefficient is represented by ID
Output to CT unit 104. The IDCT unit 104 converts the DCT coefficient of the macroblock from the inverse quantizer 103 into an inverse DCT.
T, and supplies it to the arithmetic unit 105.

The arithmetic unit 105 is supplied with the output data of the motion compensator 107 in addition to the output data of the IDCT unit 104. That is, the motion compensator 107 converts the already decoded image data stored in the frame memory 106 into I
It is read out according to the motion mode and the prediction mode from the VLC unit 102, and the arithmetic unit 1
05. The arithmetic unit 105 outputs the output data (prediction residual (difference value)) of the IDCT unit 104 and the motion compensator 10
7, the original image data is decoded. The decoded image data is supplied to and stored in the frame memory 106. If the output data of the IDCT unit 104 is intra-coded, the output data passes through the arithmetic unit 105 and is supplied to the frame memory 106 as decoded image data and stored as it is. You.

The decoded image data stored in the frame memory 106 is used as reference image data of image data to be decoded thereafter. Further, the decoded image data is
As an output reproduction image, for example, it is supplied to a display (not shown) or the like and displayed.

In MPEG1 and MPEG-2, B pictures are not used as reference image data, and are not stored in the frame memory 41 (FIG. 53) or 106 (FIG. 54) in each of the encoder and the decoder.

[0030]

FIG. 53 and FIG. 54 described above
Are compliant with the MPEG1 / 2 standard, but currently, a VO (Video Ob) which is a sequence of objects such as an object constituting an image is used.
ject), the encoding is performed in units of ISO-IEC
MPEG / JTC1 / SC29 / WG11
(Moving Picture Experts Group) 4 is being standardized.

Since MPEG4 has been standardized mainly for use in the field of communications, the GOP (Group Of Picture) defined in MPEG1 / 2 was initially MPEG
4, it is expected that efficient random access will be difficult when MPEG4 is used for storage media.

For this reason, the applicant of the present application has proposed the introduction of a GOV (Group Of VOP) corresponding to a GOP defined by MPEG1 / 2 in order to enable efficient random access. No. 80758, and this GOV was introduced in MPEG4.

On the other hand, in MPEG4, image data is hierarchized into two or more hierarchies, and flexible scalable encoding / decoding using images in each hierarchy is possible.

By the way, MPEG4 does not currently define the correspondence between the GOVs of the image data of each layer, and therefore, it is possible to insert the GOV independently for each layer. However, since the image data of each layer is not independent, when GOVs are inserted independently for each layer, it is expected that the coding efficiency may deteriorate.

The present invention has been made in view of such a situation, and is intended to prevent the coding efficiency from deteriorating.

[0036]

An image coding apparatus according to the present invention converts a sequence of objects constituting an image into lower-order sequences.
Layering means for layering into a first layer which is a layer and a second layer which is a higher layer; and a first or second layer of a sequence of objects which is output by the layering means is composed of one or more groups. And encode them, respectively, in the first or second
Encoding means for including, in the group of the first layer, a second-precision first display time representing the display time of the first displayed object in second precision, and second-precision first display for each of the first or second layer objects as a reference time, and a adding means for adding second resolution relative time information representing a second resolution of the display time of the object, adding means
Is used to assign second-accuracy relative time information to objects in the second layer.
At the time of addition , based on the display time difference between adjacent objects in the display order,
It is characterized in that the added second precision relative time information is reset.

According to the image encoding method of the present invention , a sequence of objects constituting an image is converted into a first hierarchical level which is a lower hierarchical level.
Layer and a layering layer for layering into a second layer which is a higher layer.
Step and the first or second of the sequence of objects
The hierarchy of one or more respective encoding in groups, the groups of the first or second layer, that including second resolution first display time representing a display time of an object to be the first in the second resolution, respectively Encoding step ;
Alternatively, an additional step of adding second-accuracy relative time information representing second accuracy of the display time of the object based on the second-accuracy first display time to each object of the second hierarchy.
Tsu and a flop, in addition step, the second hierarchy o
When adding second resolution relative time information object, and characterized in that the reset based on the difference between the display time of an object adjacent to each other in display order, the second resolution relative time information added to the object in the second layer I do.

According to the image decoding apparatus of the present invention , the sequence of objects constituting an image is converted into a lower layer of a first layer.
And a second layer, which is a higher layer, and encodes the first or second layer of the object sequence into one or more groups, and encodes the first or second layer into a group of the first or second layer. The display time of the object based on the second-precision first display time is given to each of the objects in the first or second hierarchy, including the second-precision first display time representing the display time of the first displayed object in second precision. Adds second-precision relative time information indicating second precision
And second-second relative time information is stored in the second layer object.
When adding, objects that are adjacent in the display order
Based on the display time difference, the objects in the second layer
It is characterized by comprising receiving means for receiving an encoded bit stream obtained by resetting the added second precision relative time information, and decoding means for decoding the encoded bit stream.

According to the image decoding method of the present invention , a sequence of objects constituting an image is converted into a first layer which is a lower layer.
And a second layer, which is a higher layer, and encodes the first or second layer of the object sequence into one or more groups, and encodes the first or second layer into a group of the first or second layer. The display time of the object based on the second-precision first display time is given to each of the objects in the first or second hierarchy, including the second-precision first display time representing the display time of the first displayed object in second precision. Adds second-precision relative time information indicating second precision
And second-second relative time information is stored in the second layer object.
When adding, objects that are adjacent in the display order
Based on the display time difference, the objects in the second layer
The present invention is characterized in that an encoded bit stream obtained by resetting the added second precision relative time information is received, and the encoded bit stream is decoded.

According to the recording medium of the present invention , a sequence of objects forming an image is converted into a first layer, which is a lower layer,
The object sequence is hierarchized into a second hierarchy, which is a higher hierarchy, and the first or second hierarchy of the object sequence is divided into one or more groups and encoded respectively, and the first or second hierarchy is firstly grouped into groups. Each of the objects in the first or second hierarchy includes a second-precision head display time representing the display time of the displayed object in second precision.
Adds second-precision relative time information that represents the second precision of the display time of the object, based on the first display time of second precision ,
Add second-accuracy relative time information to objects in the second hierarchy
Display objects that are adjacent in the display order
Based on the time difference, added to the second layer object
The encoded bit stream obtained by resetting the second precision relative time information is recorded.

Picture coding apparatus and picture coding of the present invention
The method comprises the steps of:
The first hierarchy is a lower hierarchy, and the first hierarchy is a higher hierarchy.
Layered into two levels, the first
The first or second level is divided into one or more groups
Each is encoded. Further, the first or second hierarchy
Table of objects initially displayed in a group
The second display time, which is the second
Each of the objects in the first or second hierarchy
To the object based on the second display time
Seconds relative time information indicating the seconds accuracy of the display time of the event is added.
Is done. Then, the second-precision phase is added to the object of the second hierarchy.
When time-to-time information is added, adjacent objects in display order
Based on the display time difference between
The relative second time information added to the object is reset.
Is

[0042]

In the image decoding apparatus and the image decoding method according to the present invention , a sequence of objects constituting an image
To a first hierarchy, which is a lower hierarchy, and a second hierarchy, which is an upper hierarchy.
Hierarchized into the first and second sequence of objects
Separates the second level into one or more groups,
To the first or second level group,
Seconds that represent the display time of the object displayed in
The first or second floor, including the precision start display time, respectively.
Each layer object is based on the second-precision first display time.
Represents the second precision of the display time of the object
Second-level relative time information is added, and objects in the second layer are added.
When adding second precision relative time information to the
Based on the difference between the display times of the objects
Second-accuracy relative time information to be added to the object of the second hierarchy
Bit stream obtained by resetting
The frame is received and decoded .

[0044]

In the recording medium of the present invention , an image
The sequence of objects to be
Hierarchies into a hierarchy and a second hierarchy, which is a higher hierarchy,
The first or second layer of the sequence
And encode them, respectively, in the first or second
Objects that appear first in a group in the hierarchy
The first display time in seconds, which represents the display time of the
Each object in the first or second hierarchy, including
In addition, the object based on the second display time
With relative second time information indicating the second accuracy of the display time of the event.
In addition, the second-level object has second-accuracy relative time information.
When adding, objects that are adjacent in the display order
Based on the display time difference of
By resetting the relative second time information added to
The resulting encoded bit stream is recorded.

[0046]

FIG. 1 shows a configuration example of an embodiment of an encoder to which the present invention is applied. This encoder basically performs processing conforming to the MPEG4 standard.

The image (moving image) data to be encoded is V
It is input to the O (Video Object) component 1 and the VO component 1
In, for each object constituting an image to be input thereto, the sequence in which VO is constituted, is output to the VOP configuring unit 2 ₁ to 2 _N. That is, when N pieces of VO # 1 to VO # N are configured in the VO constructing unit 1, the N pieces of VO # 1 to VO # N are output to the VOP configuring unit 2 ₁ to 2 _N.

Specifically, for example, when the image data to be encoded is composed of a sequence of independent background F1 and a sequence of foreground F2, the VO constructing unit 1 converts the sequence of foreground F2 to VO As # 1, VOP
And outputs to the configuration unit 2 _1, the sequence of the background F1, as VO # 2, and outputs the VOP constructing unit 2 _2.

When the image data to be coded is, for example, the background F1 and the foreground F2 already synthesized, the VO constructing unit 1 divides the image into regions according to a predetermined algorithm. Background F1 and foreground F
2 and VO as each sequence
To the corresponding VOP constituent unit 2 _n (where n = 1, 2,.
.., N).

The VOP constructing unit 2 _n constructs a VOP (VO Plane) from the output of the VO constructing unit 1. That is, for example,
An object is extracted from each frame, and, for example, a minimum rectangle (hereinafter, appropriately referred to as a minimum rectangle) surrounding the object is set as a VOP. At this time, the VOP constructing unit 2 _n
The VOP is configured so that the number of horizontal and vertical pixels is, for example, a multiple of 16. After constructing the VOP, the VO composing unit 2 _n outputs the VOP to the VOP encoding unit 3 _n .

Further, the VOP constructing unit 2 _n includes size data (VOP size) representing the size (eg, horizontal and vertical length) of the VOP, and the position of the VOP in the frame (eg, the upper left corner of the frame). And the offset data (VOP offset) representing the coordinates (where と す_る is the origin) are also supplied to the VOP encoder 3n.

The VOP encoding unit 3 _n outputs the output of the VOP constructing unit 2 _n to, for example, MPEG or H.264. H.263 and the like, and the resulting bit stream is output to the multiplexing unit 4. The multiplexing unit 4 includes a VO
The bit streams from the P encoders 3 _{1 to} 3 _N are multiplexed, and the resulting multiplexed data is transmitted via, for example, a terrestrial wave, a satellite line, a CATV network, or another transmission path 5, or For example, the information is recorded on a recording medium 6 such as a magnetic disk, a magneto-optical disk, an optical disk, a magnetic tape, or the like.

Here, VO and VOP will be described.

VO is a sequence of each object (object) constituting the composite image when a sequence of the composite image exists, and VOP means VO at a certain time. That is, for example, if there is a composite image F3 composed by combining the images F1 and F2, the image F1 or F2 arranged in time series is a VO, and the image F1 or F2 at a certain time.
Are VOPs. Therefore, a VO can be regarded as a set of VOPs of the same object at different times.

For example, if the image F1 is set as the background and the image F2 is set as the foreground, the composite image F3 is formed by using the key signals for extracting the image F2 and the images F1 and F2.
2, the VOP of the image F2 in this case includes not only the image data (luminance signal and color difference signal) constituting the image F2 but also its key signal as appropriate.

The sequence of image frames (image frames) does not change in both size and position, but VO
The size and position may change. That is, the same VO
May be different in size and position depending on the time.

More specifically, FIG. 2 shows an image F1 as a background.
And a composite image including the foreground image F2.

The image F1 is, for example, a photograph of a certain natural scenery, and the sequence of the entire image is one VO (VO # 0). Also, the image F2
Is an image of a person walking, for example, and the smallest rectangular sequence surrounding the person is one V
O (referred to as VO # 1).

In this case, since VO # 0 is a landscape image, basically, both its position and size do not change, similarly to a normal image frame. On the contrary,
Since VO # 1 is an image of a person, the size and position of the VO # 1 change when the person moves left and right or moves forward or backward in the drawing. Therefore, FIG.
Represents VO # 0 and VO # 1 at the same time, but the position and size of VO may change over time.

Therefore, the VOP encoding unit 3 _n in FIG. 1 outputs, to the output bit stream, information on the position (coordinate) and size of the VOP in a predetermined absolute coordinate system, in addition to the data obtained by encoding the VOP. Has been made to include. In FIG. 2, a vector indicating the position of the VOP (image F1) at a certain time, which constitutes VO # 0, is denoted by O
ST0 and V of VO # 1 at the same time
OST1 is a vector representing the position of OP (image F2);
Each is represented.

Next, FIG. 3, scalability, shows a configuration example of VOP encoding unit 3 _n of FIG.
That is, in MPEG4, scalable encoding method for realizing scalability corresponding to different image sizes and frame rates is introduced, the VOP encoding unit 3 _n shown in FIG. 3, be implemented such scalability It has been made possible.

The VOP (image data) from the VOP constructing unit 2 _n , its size data (VOPsize), and offset data (VOP offset) are all supplied to the image hierarchical unit 21.

The image layering unit 21 generates image data of one or more layers from the VOP (performs one or more layers of the VOP). That is, for example, when performing spatial scalability encoding, the image layering unit 21 outputs the image data input thereto as it is as the image data of the upper layer (upper layer), and Is reduced (decreased in resolution) by thinning out the number of pixels constituting, and this is output as image data of a lower layer (lower layer).

It is also possible to use the input VOP as data of a lower layer, increase the resolution of the VOP by some method (increase the number of pixels), and use this as data of an upper layer. It is.

The number of layers can be set to one, but in this case, scalability is not realized. Note that, in this case, the VOP encoding unit 3 _n includes, for example, only the lower layer encoding unit 25.

Further, although the number of layers can be three or more, the case of two layers will be described here for simplicity.

For example, when performing temporal scalability (temporal scalability) encoding, the image hierarchical unit 21 alternately outputs image data as lower layer or upper layer data, for example, according to time. That is, for example, the image layering unit 21 determines that VOPs constituting a certain VO are VOP0, VOP1, VOP
2, VOP3,...,
OP0, VOP2, VOP4, VOP6,... As lower layer data, and VOP1, VOP3,
VOP5, VOP7,... Are output as upper layer data. In the case of temporal scalability, the data whose VOPs have been thinned out in this way are merely data of the lower layer and the upper layer, and the image data is not enlarged or reduced (resolution conversion) (however, the resolution is not changed). , It is also possible to do).

Further, the image hierarchical unit 21 can, for example,
When performing encoding of R (Signal to Noise Ratio) scalability, input image data is output as it is as data of an upper layer or a lower layer. That is, in this case, the image data of the lower layer and the image data of the upper layer are the same data.

Here, for the spatial scalability when encoding is performed for each VOP, for example, the following three types can be considered.

That is, for example, as a VOP, FIG.
Assuming that a composite image composed of the images F1 and F2 as shown in FIG. 4 is input, the first spatial scalability becomes the entire input VOP as shown in FIG. 4 (FIG. 4A).
Is the upper layer (EnhancementLayer), and the reduced VOP (FIG. 4B) is the lower layer (Base Layer).

As shown in FIG. 5, the second spatial scalability is such that some objects (FIG. 5A (here, a portion corresponding to image F2)) constituting the input VOP are used. Extract (Note that such extraction is
For example, the processing is performed in the same manner as in the case of the VOP composing unit 2 _n , so that the object extracted thereby can also be considered as one VOP.) (FIG. 5B) is the lower layer.

Further, the third spatial scalability is:
As shown in FIGS. 6 and 7, an object (VOP) constituting the input VOP is extracted, and an upper layer and a lower layer are generated for each object. FIG. 6 shows a background (image F1) constituting the VOP of FIG.
FIG. 7 shows a case where an upper layer and a lower layer are generated from the foreground (image F2) constituting the VOP of FIG. 2.

Which of the above scalabilities is to be used is determined in advance, and the image hierarchy unit 2
1 hierarchically arranges VOPs so that encoding based on the predetermined scalability can be performed.

Further, the image layering section 21 generates a lower layer and an upper layer based on the VOP size data and offset data (hereinafter referred to as initial size data and initial offset data, respectively, as appropriate) input thereto. The offset data indicating the position of the VOP in the predetermined absolute coordinate system and the size data indicating the size are calculated (determined).

Here, V of the lower layer and the upper layer
A method for determining OP offset data (position information) and size data will be described, for example, taking the case of performing the above-described second scalability (FIG. 5) as an example.

In this case, the offset data FPOS_B of the lower layer is obtained by enlarging (upsampling) the image data of the lower layer based on the difference between the resolution of the lower layer and the resolution of the upper layer, as shown in FIG.
That is, when the image of the lower layer is enlarged, the magnification of the lower layer image matches the size of the image of the upper layer (the reciprocal of the reduction ratio when the image of the lower layer is generated by reducing the image of the upper layer). (Hereinafter, appropriately referred to as magnification FR), the offset data in the absolute coordinate system of the enlarged image is determined so as to match the initial offset data. Also, the size data FSZ_ of the lower layer
Similarly, B is determined such that the size data of the enlarged image obtained when the image of the lower layer is enlarged by the magnification FR matches the initial size data. That is, the offset data FPOS_B or the size data FSZ_B is determined so as to be equal to the respective FR times or to match the initial offset data or the initial size data.

On the other hand, the offset data FP of the upper layer
OS_E is, for example, as shown in FIG. 8B, a minimum rectangle (VO) surrounding an object extracted from the input VOP.
For example, the coordinates of the upper left vertex of P) are obtained based on the initial offset data, and are determined to this value. Also, the size data FPOS_E of the upper layer is determined to be, for example, the horizontal and vertical lengths of the minimum rectangle surrounding the object extracted from the input VOP.

Therefore, in this case, the offset data FPOS_B and the size data FPOS_B of the lower layer
Is converted according to the magnification FR (the offset data FPOS_B or the size data FPOS_B after the conversion).
Are respectively referred to as conversion offset data FPOS_B or conversion size data FPOS_B), and an image frame having a size corresponding to the conversion size data FSZ_B is considered at a position corresponding to the conversion offset data FPOS_B in the absolute coordinate system. F is the image data of the layer
While disposing an enlarged image by R times (FIG. 8)
(A)), the offset data FPOS_E and the size data FPOS of the upper layer in the absolute coordinate system.
If the image of the upper layer is similarly arranged according to _E (FIG. 8B), the pixels constituting the enlarged image and the pixels constituting the image of the upper layer have the same position at the same position. Will be placed in That is, in this case, for example, in FIG.
(B)) and the person portion in the enlarged image (FIG. 8A) are arranged at the same position.

Similarly, in the first and third scalabilities, the corresponding pixels constituting the enlarged image of the lower layer and the image of the upper layer are arranged at the same position in the absolute coordinate system. Then, offset data FPOS_B and FPOS_E and size data FSZ_B and FSZ_E are determined.

Returning to FIG. 3, the image data and offset data F of the upper layer generated by the image
The POS_E and the size data FSZ_E are delayed by a processing time in a lower layer encoding unit 25 described later in a delay circuit 22 and supplied to an upper layer encoding unit 23. The lower layer image data, offset data FPOS_B, and size data FSZ_B are supplied to the lower layer encoding unit 25. Also, the magnification FR
Is supplied to the upper layer encoding unit 23 and the resolution conversion unit 24 via the delay circuit 22.

The lower layer encoding section 25 encodes the image data of the lower layer, and the resulting encoded bit stream includes the offset data FPOS_B and the size data FSZ_B and supplies them to the multiplexing section 26. .

The lower layer coding section 25 locally decodes the coded bit stream, and outputs image data of the lower layer as a result of the local decoding to the resolution conversion section 24. The resolution conversion unit 24 includes the lower layer encoding unit 2
The image data of the lower layer from 5 is enlarged (or reduced) according to the magnification FR to restore the original size, and the enlarged image obtained by this is output to the upper layer encoding unit 23.

On the other hand, the upper layer encoding section 23 encodes the image data of the upper layer and adds the offset data FPOS_ to the resulting encoded bit stream.
E and the size data FSZ_E are included and supplied to the multiplexing unit 26. In the upper layer encoding unit 23, the encoding of the upper layer image data is performed using the enlarged image supplied from the resolution conversion unit 24 as a reference image, as described later.

The multiplexing section 26 includes the upper layer coding section 2
3 and the output of the lower layer coding unit 25 are multiplexed and output.

The lower layer encoding section 25 supplies the upper layer encoding section 23 with the lower layer size data FSZ_B, offset data FPOS_B, motion vector MV, flag COD, and the like. The conversion unit 23 performs processing while referring to these data as necessary. The details will be described later.

Next, FIG. 9 shows a detailed configuration example of the lower layer coding unit 25 of FIG. Note that, in the figure, parts corresponding to the case in FIG. 53 are denoted by the same reference numerals. That is, the lower layer encoding unit 25 is basically configured similarly to the encoder in FIG.

The image data from the image layering unit 21 (FIG. 3), that is, the VOP of the lower layer is supplied to and stored in the frame memory 31 as in the case of FIG.
In the motion vector detector 32, a motion vector is detected for each macroblock.

However, the motion vector detector 32 of the lower layer encoder 25 is supplied with the size data FSZ_B and the offset data FPOS_B of the VOP of the lower layer, where the size data FSZ_B and the offset data FPOS_B are supplied. A motion vector of a macroblock is detected based on the offset data FPOS_B.

That is, as described above, since the size and the position of the VOP change depending on the time (frame),
In detecting the motion vector, it is necessary to set a coordinate system serving as a reference for the detection and to detect a motion in the coordinate system. Therefore, here, the motion vector detector 32 uses the above-described absolute coordinate system as a reference coordinate system, and uses the absolute coordinate system as a reference in accordance with the size data FSZ_B and the offset data FPOS_B. Place a VOP to
A motion vector is detected.

The detected motion vector (MV)
Is supplied to the VLC unit 36 and the motion compensator 42 together with the prediction mode, and is also supplied to the upper layer coding unit 23 (FIG. 3).

Also, when performing motion compensation, it is necessary to detect the motion in the reference coordinate system, as described above. Therefore, the size data FSZ_B and the offset data FPOS_
B is supplied.

The VOP (macroblock of) the VOP in which the motion vector is detected is converted into a quantization coefficient and supplied to the VLC unit 36 as in the case of FIG. The VLC unit 36 also has a quantization coefficient,
The quantization step, the motion vector, and the prediction mode are supplied, and the size data FS from the image layering unit 21 is supplied.
Z_B and offset data FPOS_B are also provided, where all of these data are variable length coded.

The VOP (macroblock) of which the motion vector has been detected is coded as described above, and also locally decoded as in FIG. 53 and stored in the frame memory 41. This decoded image is used as a reference image as described above,
(FIG. 3).

Note that in MPEG4, MPEG1
Unlike B and B, since a B picture (B-VOP) is also used as a reference picture, the B picture is also locally decoded and stored in the frame memory 41 (however, at this time, the B picture is Only the upper layer is used as a reference image.)

On the other hand, the VLC unit 36 determines whether or not the macroblocks of the I, P, and B pictures (I-VOP, P-VOP, B-VOP) are to be skipped macroblocks, and indicates the result of the determination. Flag COD, MOD
Set B. The flags COD and MODB are also transmitted after being variable-length coded. Further, the flag COD
Is also supplied to the upper layer encoding unit 23.

Next, FIG. 10 shows an example of the configuration of the upper layer coding section 23 of FIG. Note that, in the figure, parts corresponding to those in FIG. 9 or FIG. 53 are denoted by the same reference numerals. That is, the upper layer encoding unit 23
Except that a frame memory 52 is newly provided, the configuration is basically the same as that of the lower layer encoding unit 25 in FIG. 9 or the encoder in FIG.

The image data from the image layering unit 21 (FIG. 3), that is, the VOP of the upper layer is supplied to and stored in the frame memory 31 as in the case of FIG.
In the motion vector detector 32, a motion vector is detected for each macroblock. In this case,
As in the case of FIG. 9, the motion vector detector 32 includes, in addition to the VOP of the upper layer, its size data FSZ.
_E and offset data FPOS_E, and the motion vector detector 32, based on the size data FSZ_E and the offset data FPOS_E, outputs the VOP of the upper layer in the absolute coordinate system in the same manner as described above. The arrangement position is recognized, and the motion vector of the macroblock is detected.

Here, the motion vector detector 32 in the upper layer encoding section 23 and the lower layer encoding section 25
Then, as in the case described with reference to FIG. 53, the VOP is processed according to a predetermined sequence that is set in advance. The sequence is set as follows, for example, here.

That is, in the case of spatial scalability, as shown in FIG. 11A or FIG.
The VOP of the upper layer or the lower layer is, for example, P,
.. Or I, P, P, P,..., Pictures (VOP).

Then, in this case, the P picture (P-VOP) which is the first VOP (in display order) of the upper layer is
For example, encoding is performed using a lower layer VOP at the same time (here, an I picture (I-VOP)) as a reference image. In addition, the second and subsequent VOs of the upper layer
The B picture (B-VOP) which is P is coded using, for example, the VOP of the immediately preceding upper layer and the VOP of the lower layer at the same time as the reference image. That is, here, the B picture in the upper layer is used as a reference image when encoding another VOP, like the P picture in the lower layer.

Note that, for the lower layer, for example, M
PEG 1 or 2, or H.264. Encoding is performed as in the case of H.263.

Since the SNR scalability is considered to be when the magnification FR in spatial scalability is 1, it is processed in the same manner as in the case of spatial scalability described above.

In the case of temporal scalability, that is, for example, as described above, VO is VOP0, VO
P1, VOP2, VOP3,...
, VOP3, VOP5, VOP7,... Are upper layers (FIG. 12A), and VOP0, VOP2, VO
When P4, VOP6,... Are the lower layers (FIG. 12B), as shown in FIG. 12, the VOPs of the upper layer or the lower layer are, for example, B, B,
B,... Or I, P, P, P,.
OP).

In this case, the first VOP1 (B picture) of the upper layer (in display order) is encoded using, for example, VOP0 (I picture) and VOP2 (P picture) of the lower layer as reference images. You. The second VOP3 (B picture) of the upper layer is, for example, the VOP1 of the upper layer coded immediately before as the B picture and the next time (frame) of VOP3.
Is encoded using the VOP4 (P picture) of the lower layer, which is the image in, as a reference image. Similarly to the VOP3, the third VOP5 (B picture) of the upper layer is, for example, the VOP3 of the upper layer coded immediately before the B picture and the lower layer which is an image at the time (frame) next to the VOP5. VOP6 (P
(Picture) as a reference image.

As described above, for a VOP of a certain layer (here, an upper layer), another layer (a scalable layer) (here, a lower layer) is used as a reference image for encoding P and B pictures. VOP can be used. As described above, when a VOP of another layer is used as a reference image to encode a VOP of a certain layer, that is, here, a VOP of a lower layer is referred to for predictive encoding of a VOP of an upper layer. When used as an image, the motion vector detector 32 of the upper layer encoding unit 23 (FIG. 10) outputs a flag ref_
A layer_id (when there are three or more layers, the flag ref_layer_id indicates a layer to which a VOP used as a reference image belongs) is set and output.

Further, the motion vector detector 32 of the upper layer encoding unit 23 sets a flag ref_
According to the layer_id, a flag ref_select__ indicating which layer VOP is to be used as a reference image for forward prediction coding or backward prediction coding, respectively.
A code (reference image information) is set and output.

That is, for example, the upper layer (Enhancement
Layer) P picture is decoded immediately before (local decoding)
If the encoding is performed using a VOP belonging to the same layer as the reference image, the flag ref_sel
ect_code is set to “00”. In addition, the P picture that is displayed immediately before and belongs to a different layer (here, a lower layer) (Reference Layer)
When encoding is performed using the OP as a reference image, the flag ref_select_code is set to “01”.
Further, when a P picture is encoded using a VOP that is displayed immediately after that and belongs to a different layer as a reference image, a flag ref_select_code is used.
e is set to “10”. When a P picture is encoded using a VOP of a different layer at the same time as the reference picture as a reference image, the flag ref_select is used.
_Code is set to “11”.

On the other hand, for example, a BOP in an upper layer uses a VOP of a different layer at the same time as a reference picture for forward prediction, and decodes immediately before the VOP belonging to the same layer. Is encoded using as a reference image for backward prediction,
The flag ref_select_code is set to “00”. In addition, a B-picture of an upper layer uses a VOP belonging to the same layer as a reference image for forward prediction, and displays a VOP belonging to a different layer, which is displayed immediately before and belongs to a different layer, as a reference image for backward prediction. When the encoding is performed by using the flag ref_select
_Code is set to “01”. Further, the BOP of the upper layer is decoded immediately before and uses the VOP belonging to the same layer as the reference image for forward prediction, and the VOP belonging to a different layer displayed immediately after that and belonging to a different layer is backward. If the encoding is performed using a reference image for prediction, the flag ref_select_c is used.
mode is set to “10”. Further, a VOP belonging to a different layer, which is displayed immediately before the B picture of the upper layer and belongs to a different layer, is used as a reference image for forward prediction,
In the case where encoding is performed using a VOP displayed immediately after that and belonging to a different layer as a reference image for backward prediction, a flag ref_select_code is used.
Is set to “11”.

Here, the method of predictive coding described with reference to FIGS. 11 and 12 is an example,
Which layer and which VOP is used as a reference image for backward prediction encoding or bidirectional prediction encoding is as follows.
For example, it can be set freely within the range described above.

In the above case, for convenience,
Although the terms “spatial scalability”, “temporal scalability”, and “SNR scalability” are used, by the flag ref_select_code,
When setting a reference image used for predictive coding, spatial scalability, temporal scalability, SNR
It is difficult to clearly distinguish scalability.
That is, to put it the other way around, the flag ref_select_co
By using de, it is not necessary to distinguish scalability as described above.

Here, associating the above-mentioned scalability with the flag ref_select_code is as follows, for example. That is, for the P picture, the flag ref_select_code is set to “1”.
Since the case of “1” is a case where the VOP of the layer indicated by the flag ref_layer_id at the same time is used as a reference image (a reference image for forward prediction), this corresponds to spatial scalability or SNR scalability. Then, the flag ref_select_
Except when the code is “11”, it corresponds to temporal scalability.

For a B picture, the flag re
Since the case where f_select_code is “00” is the case where the VOP of the layer indicated by the flag ref_layer_id at the same time is used as a reference image for forward prediction, this corresponds to spatial scalability or SNR scalability. Except when the flag ref_select_code is “00”, it corresponds to temporal scalability.

Note that, for predictive coding of the VOP of the upper layer, a different layer (here, the lower layer) is used.
When the VOP at the same time is used as a reference image, there is no motion between the two and the motion vector is always 0 ((0, 0)).

Returning to FIG. 10, the motion vector detector 32 of the upper layer coding unit 23 sets the flag re
f_layer_id and ref_select_c
mode is set, and the motion compensator 42 and the VLC unit 36
Supplied to

In the motion vector detector 32, the flags ref_layer_id and ref_select
According to t_code, a motion vector is detected not only by referring to the frame memory 31 but also by referring to the frame memory 52 as necessary.

The frame memory 52 is supplied with the locally decoded enlarged image of the lower layer from the resolution converter 24 (FIG. 3). That is, in the resolution conversion unit 24, the locally decoded VOP of the lower layer
Is enlarged (interpolated) by, for example, a so-called interpolation filter or the like, whereby the enlarged image obtained by multiplying the VOP by FR times, that is, the same size as the VOP of the upper layer corresponding to the VOP of the lower layer is enlarged. An image is generated and supplied to the upper layer encoding unit 23. In the frame memory 52, the resolution conversion unit 2
4 is stored.

Therefore, when the magnification FR is 1, the resolution conversion unit 24 does not perform any processing on the locally decoded VOP from the lower layer encoding unit 25 without any processing. 23.

The motion vector detector 32 is supplied with the size data FSZ_B and the offset data FPOS_B from the lower layer encoding unit 25, and is supplied with the magnification FR from the delay circuit 22 (FIG. 3). When the motion vector detector 32 uses the enlarged image stored in the frame memory 52 as a reference image, that is, the motion vector detector 32 uses the V
When a lower layer VOP at the same time as the OP is used as a reference image (in this case, the flag ref_select is used).
t_code is set to “11” for a P picture,
(The picture is set to “00”), and the size data FSZ_B and the offset data FPOS_B corresponding to the enlarged image are multiplied by the magnification FR. Then, based on the multiplication result, the position of the enlarged image in the absolute coordinate system is recognized, and a motion vector is detected.

The motion vector detector 32 is supplied with the motion vector of the lower layer and the prediction mode, and is used in the following case.
That is, for example, the motion vector detection unit 32 sets the flag ref_select_
When the code is “00”, the magnification FR is 1
, Ie, when the SNR is scalable (however, in this case, since the VOP of the upper layer is used for predictive coding of the upper layer, the SN
The R scalability is different from that specified in MPEG2). Since the upper layer and the lower layer are the same image, the predictive coding of the B picture of the upper layer
The motion vector and the prediction mode of the image at the same time of the lower layer can be used as they are. Therefore, in this case, the motion vector detection unit 32 does not particularly perform processing on the B picture of the upper layer, and adopts the motion vector and the prediction mode of the lower layer as they are.

In this case, upper layer coding section 23
Then, from the motion vector detector 32 to the VLC unit 36,
The motion vector and prediction mode are not output (and therefore not transmitted). This is because the receiving side can recognize the motion vector and the prediction mode of the upper layer from the decoding result of the lower layer.

As described above, the motion vector detector 32
Detects a motion vector using an enlarged image as a reference image in addition to the VOP of the upper layer, and sets a prediction mode that minimizes a prediction error (or variance) as described with reference to FIG. . Also, the motion vector detector 3
2 is, for example, a flag ref_select_code
And ref_layer_id and other necessary information are set and output.

In FIG. 10, lower layer coding section 2
5, a flag COD indicating whether a macroblock constituting an I or P picture in a lower layer is a skip macroblock is determined by the motion vector detector 3
2, the VLC unit 36, and the motion compensator 42.

The macroblock in which the motion vector is detected is coded in the same manner as described above.
The LC unit 36 outputs a variable-length code as a result of the encoding.

The VLC unit 36 of the upper layer coding unit 23 sets and outputs flags COD and MODB as in the case of the lower layer coding unit 25. Here, the flag COD indicates whether the macroblock of the I or P picture is a skip macroblock, as described above. The flag MODB indicates whether the macroblock of the B picture is a skip macroblock. It indicates whether or not.

The VLC unit 36 has a quantization coefficient, a quantization step, a motion vector, and a prediction mode.
Magnification FR, flag ref_select_code, r
The ef_layer_id, the size data FSZ_E, and the offset data FPOS_E are also supplied. The VLC unit 36 performs variable-length coding on all of these data and outputs the data.

On the other hand, the macroblock in which the motion vector has been detected is encoded, then locally decoded as described above, and stored in the frame memory 41. Then, in the motion compensator 42, similarly to the case of the motion vector detector 32, not only the locally decoded VOP of the upper layer stored in the frame memory 41 but also the local decoding The motion compensation is performed using the VOP of the lower layer enlarged as a reference image, and a predicted image is generated.

That is, in addition to the motion vector and the prediction mode, the flag ref_select_
code, ref_layer_id, magnification FR, size data FSZ_B, FSZ_E, offset data F
POS_B and FPOS_E are supplied, and the motion compensator 42 outputs the flag ref_select.
_Code, ref_layer_id, a reference image to be motion-compensated is recognized, and when a locally decoded upper layer VOP or an enlarged image is used as the reference image, its position and size in the absolute coordinate system Is recognized based on the size data FSZ_E and the offset data FPOS_E or the size data FSZ_B and the offset data FPOS_B, and a predicted image is generated using the magnification FR as necessary.

Next, FIG. 13 shows a configuration example of an embodiment of a decoder for decoding the bit stream output from the encoder of FIG.

This decoder is supplied with an encoded bit stream provided from the encoder of FIG. 1 via the transmission path 5 or the recording medium 6. That is, the bit stream output from the encoder of FIG. 1 and transmitted via the transmission path 5 is received by a receiving device (not shown), or the bit stream recorded on the recording medium 6 is reproduced by a reproducing device (not shown). The data is reproduced by the device and supplied to the demultiplexer 71.

In the demultiplexing section 71, the coded bit stream (VS (Visual Object
Seguence)) is received. Further, the demultiplexer 71, the input bit stream, the bit stream VO # 1, VO # 2 for each VO, is separated into., Respectively, are supplied to the corresponding VOP decoding unit 72 _n. V
In the OP decoding unit 72 _n , VOPs (image data) constituting a VO are obtained from the bit stream from the demultiplexing unit 71.
The size data (VOP size) and the offset data (VOP offset) are decoded and supplied to the image reconstruction unit 73.

In the image reconstruction unit 73, the VOP decoding unit 72
_The original image is reconstructed based on the output from each of _{1 to} 72 _N. The reconstructed image is supplied to the monitor 74 and displayed, for example.

Next, FIG. 14 shows a configuration example of the VOP decoding unit 72 _n of FIG. 13 for realizing scalability.

The bit stream supplied from the demultiplexing unit 71 (FIG. 13) is input to the demultiplexing unit 91, where it is separated into an upper layer VOP bit stream and a lower layer VOP bit stream. Is done. The bit stream of the VOP of the upper layer is supplied to the upper layer decoding unit 93 after being delayed by the processing time of the lower layer decoding unit 95 in the delay circuit 92, and the bit stream of the VOP of the lower layer is This is supplied to the layer decoding unit 95.

The lower layer decoding unit 95 decodes the bit stream of the lower layer, and supplies the resulting decoded image of the lower layer to the resolution converter 94. Further, the lower layer decoding unit 95 outputs size data FS obtained by decoding the lower layer bit stream.
Information necessary for decoding the upper layer VOP, such as Z_B, offset data FPOS_B, motion vector (MV), prediction mode, and flag COD, is supplied to the upper layer decoding unit 93.

The upper layer decoding section 93 includes a delay circuit 92
The upper layer bit stream provided via
The output is decoded by referring to the outputs of the lower layer decoding unit 95 and the resolution conversion unit 94 as necessary, and the resulting decoded image of the upper layer and the size data FSZ_
E and offset data FPOS_E are output. Further, upper layer decoding section 93 outputs a magnification FR obtained by decoding the bit stream of the upper layer.
Is output to the resolution conversion unit 94. The resolution conversion unit 94 converts the decoded image of the lower layer using the magnification FR from the upper layer decoding unit 93 in the same manner as in the case of the resolution conversion unit 24 in FIG. The enlarged image obtained by this conversion is supplied to the upper layer decoding unit 93, and is used for decoding the bit stream of the upper layer as described above.

Next, FIG. 15 shows an example of the configuration of the lower layer decoding section 95 in FIG. In the figure, parts corresponding to those in the decoder in FIG. 54 are denoted by the same reference numerals. That is, the lower layer decoding unit 95 is basically configured similarly to the decoder in FIG.

The bit stream of the lower layer from the demultiplexer 91 is supplied to the buffer 101, where it is received and temporarily stored. The IVLC unit 102 appropriately reads out a bit stream from the buffer 101 according to the processing state of the subsequent block, and performs variable-length decoding on the bit stream to obtain a quantization coefficient, a motion vector, a prediction mode, a quantization step, , Size data FSZ
_B, offset data FPOS_B, and flag C
Separate OD etc. The quantization coefficient and the quantization step are supplied to the inverse quantizer 103, and the motion vector and the prediction mode are determined by the motion compensator 107 and the upper layer decoding unit 9.
3 (FIG. 14). Also, the size data FSZ
_B and offset data FPOS_B are supplied to the motion compensator 107, the image reconstruction unit 73 (FIG. 13), and the upper layer decoding unit 93, and the flag COD is supplied to the upper layer decoding unit 93.

In the inverse quantizer 103, the IDCT unit 104, the arithmetic unit 105, the frame memory 106, or the motion compensator 107, the inverse quantizer 38, the IDCT unit 39, and the arithmetic unit of the lower layer coding unit 25 shown in FIG. 40, the frame memory 41, or the motion compensator 42, the same processing is performed, whereby the lower layer VOP is decoded, and the image reconstruction unit 73, the upper layer decoding unit 93, and the resolution conversion unit 94 ( FIG. 14).

Next, FIG. 16 shows a configuration example of the upper layer decoding section 93 of FIG. Note that, in the figure, parts corresponding to those in FIG. 54 are denoted by the same reference numerals. That is, the upper layer decoding unit 93 has basically the same configuration as the decoder in FIG. 54 except that the frame memory 112 is newly provided.

The bit stream of the upper layer from the demultiplexer 91 is supplied to the IVLC unit 10 via the buffer 101.
2 is supplied. The IVLC unit 102 performs variable-length decoding on the bit stream of the upper layer to obtain a quantization coefficient,
Motion vector, prediction mode, quantization step, size data FSZ_E, offset data FPOS_E, magnification FR, flags ref_layer_id, ref_se
Select_code, COD, MODB, etc. are separated. The quantization coefficient and the quantization step are supplied to the inverse quantizer 103 as in the case of FIG. 15, and the motion vector and the prediction mode are supplied to the motion compensator 107. Further, the size data FSZ_E and the offset data FPOS_E are supplied to the motion compensator 107 and the image reconstruction unit 73 (FIG. 13), and the flags COD, MO
DB, ref_layer_id, and ref_se
The select_code is supplied to the motion compensator 107. Further, the magnification FR is supplied to the motion compensator 107 and the resolution converter 94 (FIG. 14).

In addition to the data described above, the motion compensator 107 receives the motion vector, flag COD, size data F of the lower layer from the lower layer decoding unit 95 (FIG. 14).
SZ_B and offset data FPOS_B are supplied. Also, the frame memory 11
2 is supplied with an enlarged image from the resolution conversion unit 94.

Inverse quantizer 103, IDCT unit 104, arithmetic unit 105, frame memory 106, motion compensator 10
7 or the frame memory 112, the inverse quantizer 38, IDCT unit 39,
By performing the same processing as in the arithmetic unit 40, the frame memory 41, the motion compensator 42, or the frame memory 52, the VOP of the upper layer is decoded and supplied to the image reconstruction unit 73.

Here, a VO having upper layer decoding section 93 and lower layer decoding section 95 configured as described above is provided.
In the P decoding unit 72 _n , the decoded image, the size data FSZ_E, and the offset data FPOS_E for the upper layer (hereinafter, all of them are appropriately included,
Upper layer data), a decoded image of the lower layer for the upper layer, and size data FSZ_
B and offset data FPOS_B (hereinafter referred to as lower layer data including all of them as appropriate)
The image reconstructing unit 73 reconstructs an image from the upper layer data or the lower layer data as follows, for example.

That is, for example, when the first spatial scalability (FIG. 4) is performed (when the entire input VOP is set as the upper layer and a reduced version of the entire VOP is set as the lower layer) When both the lower layer data and the upper layer data are decoded, the image reconstructing unit 73 generates a decoded image (VOP) of the upper layer having a size corresponding to the size data FSZ_E based on only the upper layer data. , At the position indicated by the offset data FPOS_E. Further, for example, when an error occurs in the bit stream of the upper layer, or when the monitor 74 supports only low-resolution images, decoding of only the lower layer data is performed. Arranges a decoded image (VOP) of an upper layer having a size corresponding to the size data FSZ_B at a position indicated by the offset data FPOS_B based on only the lower layer data.

Further, for example, when the second spatial scalability (FIG. 5) is performed (a part of the input VOP is set as an upper layer, and a reduced whole of the VOP is set as a lower layer). In the case, when both the lower layer data and the upper layer data are decoded, the image reconstructing unit 73 outputs the size data F
A decoded image of a lower layer having a size corresponding to SZ_B is
The image is enlarged according to the magnification FR, and an enlarged image is generated. Further, the image reconstructing unit 73 outputs the offset data F
POS_B is multiplied by FR, and the enlarged image is arranged at a position corresponding to the value obtained as a result. Then, the image reconstruction unit 7
No. 3 arranges the decoded image of the upper layer of the size corresponding to the size data FSZ_E at the position indicated by the offset data FPOS_E.

In this case, the decoded image portion of the upper layer is displayed with a higher resolution than the other portions.

When the decoded image of the upper layer is arranged, the decoded image and the enlarged image are combined.

Although not shown in FIG. 14 (FIG. 13), the upper layer decoding unit 93 (VOP decoding unit 72 _n ) sends the image reconstruction unit 73
The magnification FR is also supplied, and the image reconstruction unit 73 generates an enlarged image using the magnification FR.

On the other hand, when only the lower layer data is decoded when the second spatial scalability is performed, an image is reconstructed in the same manner as when the first spatial scalability is performed. .

Further, when the third spatial scalability (FIGS. 6 and 7) is performed (for each object constituting the input VOP, the entire object (object) is set as the upper layer, and (When the lower layer is obtained by thinning out), the image is reconstructed in the same manner as when the above-mentioned second spatial scalability is performed.

As described above, the offset data FPO
S_B and FPOS_E are such that the corresponding pixels constituting the enlarged image of the lower layer and the image of the upper layer are arranged at the same position in the absolute coordinate system. With this configuration, an accurate (no displacement) image can be obtained.

Next, the syntax of the coded bit stream output from the encoder shown in FIG.
The following describes an example of the Video Verification Model (Version 10.0) of the four standards (hereinafter, appropriately referred to as VM 10.0).

FIG. 17 shows the configuration of an encoded bit stream in VM 10.0.

The coded bit stream is VS (Visual
Object Sequence). here,
VS is an image sequence, and corresponds to, for example, one program or movie.

Each VS has one or more VISOs (Visual Obj
ect). Here, there are several types of VISO. That is, the VISO includes, for example, a still texture object (Still Texture Ob
ject), a face object composed of face images, and a VO (V
ideo Object). Therefore, when the encoded bit stream is that of a moving image, VISO is composed of VO.

A VO is composed of one or more VOLs (Video Object L
ayer) (when the image is not hierarchized, it is composed of one VOL, and when the image is hierarchized, it is composed of VOLs of the number of layers).

A VOL is a required number of GOVs (Group of V
OP), and the GOV is composed of one or more VOPs (Video Obs).
ject plane). In addition, GOV
In this case, VOL is not less than one VOP
It will be composed of

FIG. 18 or FIG. 19 shows the syntax of VS or VO, respectively. A VO is a bit stream corresponding to a sequence of an entire image or a part (object) of an image, and thus a VS is composed of a set of such sequences (thus, a VS is, for example, a single program). Etc.).

FIGS. 20 to 25 show the syntax of a VOL.

The VOL is a class for scalability as described above, and is identified by a number indicated by video_object_layer_id. That is, for example, the video_object_layer_id for the VOL of the lower layer is set to 0, and for example, the vi_object_layer_id for the VOL of the upper layer is set.
deo_object_layer_id is set to 1. As described above, the number of scalable layers is not limited to two, but may be any number including one or three or more.

For each VOL, whether it is the entire image or a part of the image is determined by video_object_l
Identified by ayer_shape. This video_object_layer_sha
pe is a flag indicating the shape of the VOL, and is set, for example, as follows.

That is, when the VOL is rectangular, the video_object_layer_shape is set to, for example, "00". When the VOL has a shape of an area extracted by a hard key (a binary (Binary) signal having one of 0 or 1), video_ob
ject_layer_shape is, for example, “01”. Further, a signal (MPEG4) in which the VOL can take soft keys (gray-scale) in a continuous range of 0 to 1
In this case, when represented by 8 bits)), the shape of the region extracted (when the image is synthesized using soft keys), video_object_layer_shape is
For example, “10” is set.

Here, video_object_layer_shape is set to “0”.
It is set to "0" when the VOL has a rectangular shape and the position and size of the VOL in the absolute coordinate form do not change with time, that is, are constant. In this case, the sizes (horizontal length and vertical length) are video_object_layer_width and video_object_lay
Indicated by er_height. video_object_layer_widt
h and video_object_layer_height are both 10-bit fixed-length flags. When video_object_layer_shape is “00”, they are transmitted only once first (this is as described above when video_object_layer_shape is “00”). , VOL in the absolute coordinate system are constant).

Further, whether the VOL is a lower layer or an upper layer is indicated by scalability which is a 1-bit flag. If the VOL is a lower layer, the scalability is, for example, 1; otherwise, the scalability is, for example, 0.

Further, when a VOL uses an image in a VOL other than itself as a reference image, the VOL to which the reference image belongs is represented by ref_layer_id as described above. Note that ref_layer_id is transmitted only for the upper layer.

In the VOL, hor_sampling_fac
tor_n and hor_sampling_factor_m are lower layer VOPs
And the VOP of the upper layer
The values corresponding to the length in the horizontal direction are shown. Therefore, the horizontal length (horizontal resolution magnification) of the upper layer with respect to the lower layer is expressed by the formula hor_sampling_facto.
Given by r_n / hor_sampling_factor_m.

Furthermore, in the VOL, ver_sampling_f
actor_n and ver_sampling_factor_m are the VOs of the lower layer
The value corresponding to the vertical length of P and the VO of the upper layer
The values corresponding to the vertical length of P are shown. Therefore, the vertical length (magnification of the vertical resolution) of the upper layer with respect to the lower layer is calculated by the expression ver_sampling_facto.
Given by r_n / ver_sampling_factor_m.

FIGS. 26 and 27 show the syntax of GOV.

The GOV layer is defined between the VOL layer and the VOP layer so that it can be inserted not only at the head of the coded bit stream but also at any position in the coded bit stream (FIG. 17). ). As a result, a certain VOL # 0 becomes VOP
# 0, VOP # 1, ..., VOP # n, VOP # (n + 1), ..., VOP # m
In the case of a VOP sequence such as
The OV layer is not only immediately before the first VOP # 0, but also VOP # (n + 1)
(Between VOP # n and VOP # (n + 1)). Therefore, in the encoder, the GOV layer can be inserted, for example, at a position to be randomly accessed in the encoded stream. By inserting the GOV layer, a series of VOPs constituting a certain VOL becomes , Are encoded in a plurality of groups (GOV).

As shown in FIG. 26, the GOV layer has a group start code (group_start_code), a VOP time increment resolution (VOP_time_increment_reso).
lusion), time code (time_code), closed GOP
(Closed_gop), broken links (broken_link),
VOP (Group_of_VideoObjectPlan) that constitutes GOV
e ()), next start code (next_start_code ())
Are sequentially arranged.

Next, the semantics of the GOV layer (Semantic
s), the semantics of the GOV layer are basically the same as those of the GOP layer of MPEG2, and therefore, unless otherwise specified, the MPEG2 Video standard (ISO / IEC1381
Please refer to 8-2).

First, group_start_code is 000001B8
(Hexadecimal) indicates the starting position of the GOV. VOP_time_incr
The ement_resolusion represents the number of divisions (resolution) of how many divisions are made into one second, which is an interval between synchronization points indicated by VOP_time_increment described later.

As shown in FIG. 27, the time_code is composed of time_code_hours representing the time unit of time, time_code_minutes representing the time unit, marker_bit, and time_code_seconds representing the unit of time. And of this time_code, time_code_hours, time_c
ode_minutes and time_code_seconds indicate the time at the beginning of the GOV. As a result, the time_code (second display time at the beginning of the second) of the GOV layer is the second time at the second accuracy, that is,
It represents the absolute time on the VOP sequence at which the encoding of the GOV layer has started. Specifically, time_cod
In e, the absolute display time of the VOP displayed first in the GOV is set in seconds.

In VM 10.0, time_code is set to “T
he parameters correspond to those defined in the I
EC standard publication 461 for "time and control
codes for video tape recorders ". The time code ref
ers to the first plane (in display order) after t
he GOV header ".

The marker_bit of the time_code is set to 1 so that 23 or more 0s do not continue in the encoded bit stream.

Next_start_code () gives the start position of the next GOV.

According to the GOV described above, it is possible to recognize the absolute display time of the first VOP displayed in the display order following the GOV header with second precision by the time code time_code. Become. Note that, as described above, GO
Since the time_code of the V layer is accurate to the second, a portion of the display time of the VOP with a finer accuracy is set for each VOP. This point will be described later.

As described above, the GOV layer was introduced in MPEG4 after the proposal by the present applicant.

Next, FIGS. 28 to 36 show VOPs (Vide
o Indicates the syntax of the Object Plane Class).

The VOP size (horizontal and vertical lengths) is represented by, for example, VOP_width and VOP_height having a fixed length of 10 bits. The position of the VOP in the absolute coordinate system is, for example, VOP_horizontal_spatial_mc_re of 10-bit fixed length.
f and VOP_vertical_mc_ref. Note that VOP_width
Alternatively, VOP_height represents the length of the VOP in the horizontal direction or the vertical direction, respectively, and corresponds to the size data FSZ_B or FSZ_E described above. Also, VOP_horizo
ntal_spatial_mc_ref or VOP_vertical_mc_ref is V
The horizontal and vertical coordinates (x or y coordinates) of the OP are respectively represented, and these correspond to the above-described offset data FPOS_B and FPOS_E.

VOP_width, VOP_height, VOP_horizontal_
spatial_mc_ref and VOP_vertical_mc_ref are video
It is transmitted only when _object_layer_shape is other than “00”. That is, when the video_object_layer_shape is “00”, as described above, since the size and position of the VOP are all constant, the VOP_width, VOP_height, and VOP_h
orizontal_spatial_mc_ref, and VOP_vertical_mc_re
f need not be transmitted. In this case, on the receiving side, VO
P is arranged such that its upper left vertex coincides with, for example, the origin of the absolute coordinate system.
To the video_object_layer_width of the VOL shown in FIG.
And video_object_layer_height.

[0182] In the VOP, ref_select_code
Represents an image used as a reference image, as described above.

By the way, in VM 10.0, each VOP (Video Obje
The display time of ct Plane (corresponding to the conventional Frame) is modulo_time_base and VOP_ti defined by the VOP.
It is determined as follows by me_increment (FIG. 28) and time_code (FIG. 27) defined by the GOV composed of the VOP.

That is, modulo_time_base represents the time on the local time axis of the encoder with an accuracy of one second (1000 ms (millisecond)). modulo_time_base is represented by a marker transmitted in the VOP header, and is composed of a required number of “1” and one “0”. modulo_t
The number of "1" s constituting the ime_base is the last (last most recently) (last) I-VOP,
Modulo_time_base encoded / decoded in P-VOP,
Alternatively, it indicates the accumulated time from the synchronization point (1 second accurate time) indicated by the time_code of the GOV header.

More specifically, the modulo_time_base of the VOP
Is, for example, "0", the I-V displayed immediately before
Indicates that the accumulated time from the synchronization point indicated by the modulo_time_base of the OP or P-VOP or the time_code of the GOV header is 0 second. Also, modulo_time_base
Is "10", for example, if the I-
VOP, modulo_time_base of P-VOP, or GOV
It indicates that the accumulated time from the synchronization point indicated by the time_code of the header is 1 second. Furthermore, modulo_time_ba
When se is, for example, “110”, modulo_time_base of the I-VOP or P-VOP displayed immediately before, or G
It indicates that the accumulated time of the synchronization point indicated by the time_code of the OV header is 2 seconds. As mentioned above,
The number of “1” in modulo_time_base is the number of seconds from the synchronization point indicated by the modulo_time_base of the I-VOP or P-VOP displayed immediately before or the time_code of the GOV header.

[0186] In VM 10.0, modulo_time_base is described as "This value represents the local time base".
at the one second resolution unit (1000 millisecon
ds) .It is represented as a marker transmitted in t
he VOP header. The numberof consecutive "1" follow
ed by a "0" indicates the number of seconds has el
apsed since the synchronization point marked by th
e modulo_time_baseof the last displayed I / P-VOPs b
elonging to the same VOL. There are twoexceptions,
one for the first I / P-VOP after the GOV header, a
nd the other is for B-VOPs (in display order) to t
he first I-VOP after the GOV header.For the first
I / P-VOP after the GOV header, the modulo_time_base
indicates the time relative to the time_code in t
he GOV header.For the B-VOPs prior (in display ord
er) to the first I-VOP after the GOV header, themo
dulo_time_base indicates the time relative to the
time_code in the GOV header ".

That is, modulo_time_base of a certain VOP of interest
Is the modulo of the I-VOP and P-VOP displayed immediately before
The synchronization point indicated by _time_base, that is, the relative time from the second-precision display time of the I-VOP or P-VOP displayed immediately before represents the second-precision display time of the target VOP. However, the modulo_tim of the I-VOP or P-VOP to be encoded / decoded first following the GOV header
e_base indicates the display time of the I-VOP or P-VOP with a second precision by the relative time from the time_code of the GOV header, and indicates the time before the first I / VOP encoded / decoded following the GOV header. B-VO displayed on
The modulo_time_base of P also indicates the display time of the B-VOP with a second precision by the relative time from the time_code of the GOV header.

Note that the I-VOP or P-VOP encoded / decoded first following the GOV header and the B-VOP displayed before such I-VOP are indicated by the time_code of the GOV header. The synchronization point to be changed is a modulo of the I-VOP and P-VOP displayed immediately before.
_time_base, the modulo_time_base of the VOP is what kind of V, given the synchronization point (the time of the second precision of the display time of the I-VOP and P-VOP displayed immediately before).
As for OP, I-VOP, PV displayed immediately before
It can be said that the display time of the VOP is expressed in seconds with the time from the synchronization point indicated by the modulo_time_base of the OP.

VOP_time_increment is V with an accuracy of 1 second or less.
Used to indicate the display time of the OP. I.e. time_co
Although de and modulo_time_base represent time with an accuracy of one second, precisions lower than that are expressed using VOP_time_increment. In the case of VM 10.0, the accuracy of VOP_time_increment is the same as the time_increment_resolution of the GOV layer described above.
(FIG. 26), and VOP_time_increment_resolution
Assuming that = 1000, VOP_time_increment sets the display time of the VOP to 1 ms (= 1000 milliseconds / VOP_time_incremen).
t_resolution) unit.

FIGS. 37 and 38 show time_code, modulo_time_base and VOP_time_increme based on the above definitions.
FIG. 4 is a diagram showing a relationship with nt.

In FIG. 37, VO is a GOV header,
B₁(B-VOP), I_Two(I-VOP), B_Three, B_Four, P_Five(P
-VOP) .... is composed of a VOP sequence
I have. Now, the time_code of the GOV header is the time t₀ = 0h:
12m: 43Sec (0:12:43)
And modulo_time_base is the time based on time_code
With one second precision and thus t₀+1 second, t₀+2 seconds,
.. Are represented as synchronization points. Note that FIG.
, The display order is B₁, I_Two, B_Three, B_Four, P_Five, ...
, But the encoding / decoding order is the GOV header, I_Two,
B₁, P_Five, B_Three, B _Four, ...

In FIG. 37, (FIG. 38, FIG.
VOP_time for each VOP
_increment is a number enclosed in a square, modulo_time_ba
se is indicated by a bit string surrounded by a square and double quotation marks ("), respectively. In addition, VOP_time_i
ncrement_resolution is set to 1000. Therefore,
In FIG. 37, VOP_time_i for B ₁ , I ₂ , B ₃ , B ₄ , and P ₅
ncrement is 350, 750, 150, 550, respectively
350 and VOP_time_increment_resol
Since the solution is 1000, the display time of each VOP for 1 second or less is 350 ms (350/1000 seconds) and 75, respectively.
0 ms (750/1000 seconds), 150 ms (150/10
00 seconds), 550 ms (550/1000 seconds), 350 ms
(350/1000 seconds).

Now, in FIG. 37, B ₁ is a B-VOP displayed before I ₂ which is the first I-VOP to be encoded / decoded after the GOV header is encoded / decoded.
Therefore, the reference of the display time is ti of the GOV header.
the time t ₀ is the synchronization point indicated by me_code. Now
The time_modulo_base of B ₁ is “0”, so that B ₁
Is displayed at the time t indicated by the time_code of the GOV header.
₀ , 0 + 350/1000 seconds later, that is, 0h: 12
m: 34s + 0s: 350ms = 0h: 12m: 34s: 350ms.

Next, since I ₂ is an I-VOP that is encoded / decoded first after the GOV header is encoded / decoded, the reference of the display time is time_c of the GOV header.
It is time t ₀ , which is the synchronization point indicated by ode. Now I ₂
Time_modulo_base is “0”, so the display time of I ₂ is the time t ₀ indicated by the time_code of the GOV header.
Time of 0 + 750/1000 seconds, ie 0h: 12m: 34s + 0s: 750
ms = 0h: 12m: 34s: 750ms.

Next, since B ₃ is displayed immediately after I ₂ , the reference of the display time is a synchronization point indicated by modulo_time_base of I ₂ . That is, display time of I _2, as described above, 0h: 12m: 34s: a 750 ms, Modulo_time_
base is the time 0 seconds after time t ₀ = 0h: 12m: 34Sec
0h: 12m: 34s is shown as a synchronization point. And B ₃
Is the modulo_time_base, since it is "10", the display time of B ₃ is the synchronization point marked by the modulo_time_base of I _2, 1 + 150/1000 seconds later time, namely 0
h: 12m: 34s + 1s: 150ms = 0h: 12m: 35s: 150ms. The display time of B ₄ can be obtained in the same manner as the display time of B ₃ , and the synchronization point indicated by the modulo_time_base of I ₂
Time after 1 + 550/1000 seconds, ie 0h: 12m: 34s
+ 1s: 550ms = 0h: 12m: 35s: 550ms.

[0196] Next, in order for P _5, B _3, B ₄ to be displayed on the front are both B-VOP, these modulo
synchronization point marked by the _time_base is not used for calculating the display time of P _5, I / P, which is displayed immediately before the P ₅
Synchronization point marked by the modulo_time_base of I ₂ is -VOP (0h: 12m: 34Sec) is used. FIG.
In, time_modulo_base of P ₅ is a "110", time_i
ncrement, since it is 350 as described above, the display time, the synchronization point marked by the modulo_time_base of I _{2, 2} + 350/1000 seconds later time, i.e., 0h: 12m: 34s + 2s: 350ms = 0h: 12m: 36s: 350ms.

Next, in FIG. 38, a VO is composed of a GOV header and a sequence of VOPs of B ₁ , B ₂ , I ₃ , B ₄ , P ₅ ,. Also, the ti of the GOV header
me_code is the time t ₀ = 0 as in the case of FIG.
h: 12m: 43Sec (0:12:43).

[0198] In FIG. 38, for B _1, modulo_
The time_base is “0” and the VOP_time_increment is 350. Then, B ₁ is, GOV header is encoded /
After being decoded, since it is first B-VOP displayed before I ₃ which is I-VOP which is encoded / decoded, the display time is synchronous point indicated by the time_code of the GOV header time It is calculated based on t ₀ . Therefore, the display time of B ₁ represents, as with B ₁ in FIG. 37, 0h: 12m: 34s + 0s: 350
ms = 0h: 12m: 34s: 350ms.

[0199] Next, B _2, as with B _1, after the GOV header is encoded / decoded and encoded / decoded first I
Since a B-VOP displayed before I ₃ which is -Vop, the display time is still, the GOV header time
It is calculated on the basis of the time t ₀ is the synchronization point indicated by _CODE. Now, time_modulo_base of B ₂ is at "0", VOP_tim
Since e_increment has become 750, the display time of B ₂ is the time t ₀ shown in time_code of the GOV header, 0 +
Time after 750/1000 seconds, that is, 0h: 12m: 34s + 0s:
750ms = 0h: 12m: 34s: 750ms.

Next, I ₃ is obtained by encoding / decoding GOV.
This is the I-VOP displayed first, and its modulo_time_
Since the base is “01” and the VOP_time_increment is 150, the display time is delayed by 1 + 150/1000 seconds from the synchronization point indicated by the time_code. Therefore, I ₃
Is the time 0 + 750/1000 seconds after time t ₀ , that is, 0h: 12m: 34s + 1s: 150ms = 0h: 12m: 35s: 150
ms.

[0202] Next, the B _4, synchronization point marked by the modulo_time_base of I ₃ is I / P-VOP displayed immediately before (seconds accuracy of the display time of I _3), i.e., 0h: 12m: The display time is calculated based on 35s.
Now, modulo_time_base of B ₄ is "0", VOP_time_in
Because the crement is 550, the display time is 0h: 12m: 35s
+ 0s: 550ms = 0h: 12m: 35s: 550ms

[0202] Next, the P _5, synchronization point marked by the modulo_time_base of I ₃ is I / P-VOP displayed immediately before, i.e., as with B _4, 0h: 12m: 35s
, The display time is calculated. Now, mod of P ₅
ulo_time_base is “10” and VOP_time_increment is 350
Therefore, the display time is 0h: 12m: 35s + 1s: 350ms = 0
h: 12m: 36s: 350ms.

Here, as described above, in VM 10.0, the synchronization point indicated by modulo_time_base of the I / P-VOP (which is equal to the second precision of the display time) is used for the calculation of the display time. However, the synchronization point indicated by the modulo_time_base of the B-VOP is not used. This is mainly due to the following reasons. In other words, B-VOPs are displayed as I / P
Although sandwiched between VOPs, in the encoding / decoding order, the I / PVOP sandwiching the B-VOP is encoded / decoded after being encoded / decoded. Therefore, if the display time is calculated, the B-VOP
If the synchronization point indicated by modulo_time_base is used, that is, if the second precision of the display time of the B-VOP is used, the I / P-VOP located immediately after the B-VOP in the display order is already encoded. This is because the display time of the I / P-VOP cannot be obtained until the encoding / decoding of the B-VOP is completed despite the encoding / decoding, and the processing becomes complicated. .

Next, in the VM 10.0, as described above, the GOV
Layers are used, thereby enabling efficient random access. That is, time_co of the GOV layer
The absolute departure time of GOV (however, second precision) is determined by de, and this start time and time_modulo_base, VOP_time_in
As described above, the absolute display time of each VOP included in the GOV is easily obtained by the crement. Then, by referring to the display time, it becomes possible to access a VOP in the middle of the encoded bit stream, that is, to make random access.

By the way, in VM 10.0, “The GOV layer
is an optional layer, so the bitstream can have an
y (include non) number of the GOV header, and the
frequency of the GOV header is an encoder issue.Si
nce the GOV header shall be followed by the I-VO
P. ", the insertion of the GOV layer is performed on the encoder side except that the I-VOP must be placed after the GOV header in the encoded bit stream. The number and frequency can be set freely.

That is, in VM 10.0, when hierarchization for realizing scalability is performed, G
The insertion position of the OV is not particularly defined. Therefore, for example, when the layers are hierarchized into two layers of an upper layer and a lower layer, an I-VOP is independently arranged after the header for each of the upper layer and the lower layer (however, in the coded bit stream, ) Can be constructed.

On the other hand, in MPEG4, the upper layer can encode / decode without necessarily using the I-VOP. This is because the VOP of the lower layer is basically used as a reference image for the upper layer. In MPEG4, the coding efficiency is improved by using the VOP of the lower layer as the reference image. Let me.

However, as described above, in order to use the GOV, it is necessary to add the IV after the GOV header in the encoding / decoding order.
OP must be deployed. And, since the data amount of the I-VOP is generally larger than the data amount of the P-VOP or B-VOP, the upper layer that does not have a problem even without using the I-VOP has
Having to use an OP reduces coding efficiency.

Therefore, in order to prevent a decrease in coding efficiency, the above definition defined in VM 10.0 is “The GOV
layer is an optional layer, so the bitstream can h
aveany (include non) number of the GOV header, and
the frequency of the GOVheader is an encoder issu
e.Since thr GOV header shall be followed by the I-
VOP in base layer. ", And should place a restriction that the I-VOP must be placed after the GOV header only for the lower layer, and not for the upper layer. .

The encoder shown in FIG. 1 or the decoder shown in FIG. 13 performs encoding or decoding, respectively, according to the definition after the change.

Next, in the VM 10.0, there is no particular restriction on the temporal positional relationship between the GOVs of the lower layer and the upper layer. Therefore, the first VOP or the last VOP of the GOV of the lower layer and the first VOP or the last VOP of the upper layer do not always need to coincide with each other.

However, when the insertion position of the GOV is different between the lower layer and the upper layer, when random access is made to the encoded bit stream, in order to obtain a decoded image of the lower layer referred to by the VOP of the upper layer, In some cases, it is necessary to decode a VOP of a lower layer that is not originally displayed. In this case, it is difficult to perform quick and efficient random access.

That is, #i in the display order of the lower layer
Th the I-VOP or P-VOP, respectively, I _bi
Or P _bi, and the # i-th P-VOP in the display order of the upper layer is represented as P _ei .
As shown in FIG. 39, I _b1 , P _b2 , P _b3 , P _b4 , P _b5 ,
_{I b6, P b7, P b8} , P b9, P b10, I b11, P b12,
P _{b 13,} and the lower layer composed of a ···, P _{e1,
P e2, P e3, P e4} , P e5, P e6, P e7, P e8, P e9, P
_{e10, P e11, P e12,} P e13, consider an encoded bit stream consisting of a composed upper layer in ....

In FIG. 39, the lower layer I_b1,
P_b2, P_b3, P_b4, P_b5, I_b6, P_b7, P_b8, P_b9, P
_b10, I_b11, P_b12, P_b13, ... each and the top level
Ear P _e1, P_e2, P_e3, P_e4, P_e5, P_e6, P_e7, P
_e8, P_e9, P_e10, P_e11, P_e12, P_e13, ...
This is a VOP displayed at the same time,
Upper layer P_e1Or P_e13Each is a lower layer
I_b1Or P_b13Each is encoded as a reference image
I have. That is, spatial scalability is realized.
Then, in FIG. 39, the lower layer I_b1Or P_b5Is GO
V_b1To I_b6Or P _b10Is GOV_b2And configure
And the upper layer P_e1Or P_e4Is GOV_{e 1}And P_e5
Or P_e8Is GOV_e2And P_e9Or P_e12Is GOV_e3To
Each is composed. That is, in the lower layer, five
A GOV is configured for each VOP, and four
GOV is configured for each VOP.

[0215] In this case, for example, when a random access to the beginning of GOV _e2 of the upper layer, the head of P _e5 is decoded using the decoded image of P _b5 of GOV _b1 of the lower layer as a reference picture Therefore, it is necessary to decode _Pb5 . However, P _b5 is the lower layer G
This is the VOP that constitutes OV _b1 (the last displayed VOP). Therefore, to decode P _b5 , it is necessary to sequentially decode the VOPs that constitute GOV _b1 . Until the P ₆₅ is decoded, not possible to decode the P _e5 upper layer using the P ₆₅ as a reference picture, fast random access is difficult.

Such a situation also occurs in the case of time scalability.

That is, as shown in FIG. 40, b ₁ ,
.., and e ₁ , e ₂ , e ₃ , e ₄ , e ₅ , e ₆ , e, and lower layers composed of VOPs represented by b ₂ , b ₃ , b ₄ , b ₅ ,.
_7, e _8, consider a coded bit stream of the upper layer composed of a VOP represented by .... In addition,
e ₁ , b ₁ , e ₂ , b ₂ , e ₃ , e ₄ , b ₃ , e ₅ , e ₆ , b ₄ ,
e ₇ , e ₈ , b ₅ ,... constitute a series of image sequences, and in FIG. 40, from such image sequences, b ₁ , b ₂ , b ₃ , b ₄ , b ₅ ,. Are extracted to form a lower layer, e ₁ , e ₂ , e ₃ , e ₄ , e ₅ ,
e ₆ , e ₇ , e ₈ ,... are extracted to form an upper layer.

In FIG. 40, b in the lower layer₁No
To b_ThreeIs GOV_b1And b_FourOr b₆Is GOV_b2Each
And the upper layer e₁Or e_ThreeIs GOV
_e1To e_FourOr e₆Is GOV_e2To e₇Or e₉Is GOV_e3
, Respectively. That is, lower layer and upper
GOV is configured for every three VOPs in both layers
Have been. Note that e₁Is b₁To e_TwoAnd e_ThreeIs b
_TwoTo e_FourAnd e_FiveIs b_ThreeTo e ₆And e₇Is b_FourTo e₈
Is b_FiveAre used as reference images.

[0219] In this case, for example, when a random access to the beginning of GOV _e2 of the upper layer, the head of the e ₄ is decoded using the decoded image of b ₃ of GOV _b1 of the lower layer as a reference picture First, b ₃
Need to be decrypted. However, b ₃ is displayed on the VOP (last constituting the GOV _b1 of the lower layer V
A OP), therefore, in order to decode the e ₄ is, GOV _b1
Must be sequentially decoded. And
until the decode b ₃ is not possible to decode the e ₄ of the upper layer is a purpose of the random access again, rapid random access is difficult.

Therefore, the VOP of the upper layer displayed at the same time as or immediately after the display time of the VOP displayed first of the GOV of the lower layer is set so as to be displayed first of the GOV of the upper layer. , Upper layer, GO
Divide into P. By doing so, the GOV between the upper layer and the lower layer has a one-to-one correspondence, and it is possible to prevent rapid random access from being hindered.

That is, for example, the same as in the case shown in FIG.
If upper and lower layers are configured,
And the lower layer P_b5Is the first VOP displayed
So that the lower layer GOV is configured as
As shown in FIG. _b5At the same time as
P of the upper layer shown_e5Is the first VOP displayed
Thus, the GOV of the upper layer is configured. Also examples
For example, P in the lower layer _b10Is the first displayed VOP
When the lower layer GOV is configured so that
Also, as shown in FIG._b10At the same time as the display time
P of the upper layer displayed at the moment_e10Appears first
The upper layer GOV is configured so that

[0222] Further, for example, VOP in the same manner as shown in FIG. 40, when the upper layer and the lower layer is formed, the lower layer b ₂ is initially displayed
As will be, VOP when configuring the GOV of the lower layer, as shown in FIG. 42, the upper layer of e ₃ that appears immediately after the display time of the b ₂ is first displayed
The GOV of the upper layer is configured so that Also,
For example, VOP b ₄ of the lower layer is to be displayed first
As will be, VOP when configuring the GOV of the lower layer, as shown in FIG. 42, the upper layer of e ₇ that appears immediately after the display time of the b ₄ is first displayed
The GOV of the upper layer is configured so that

[0223] For the GOV, the GOV header is
It is encoded every time, but the encoding result is placed immediately before the VOP to be encoded / decoded first in the GOV.
As the time_code of the GOV header, the display time of the VOP displayed first in the GOV is encoded with second precision.

[0224] Thus, in FIG. 41, for example, time_code becomes a second resolution of the display time of the P _b5, first displayed P _e5 to P _b5 is placed GOV header of GOV of the lower layer to be displayed first GOV GO of upper layer
Time_code placed in V header, the second resolution of the display time of P _e5. Then, in FIG. 41, the display time of P _b5 and P _e5 are the same, therefore, the time_code of P _b5 is initially placed in the GOV header of the GOV of the lower layer to be displayed,
Time_code of P _e5 is initially placed in the GOV header of a GOV of the high-level layer to be displayed the same value.

In FIG. 42, for example, the time_code placed in the GOV header of the GOV of the lower layer in which b ₂ is displayed first is the second precision of the display time of b ₂ ,
Time_code _{of 3} is placed first in the GOV header of a GOV of the high-level layer to be displayed, the second resolution of the display time of e _3.

Next, the process of constructing the GOV of the lower layer (the process of inserting the GOV header) is performed by the VLC unit 36 (hereinafter referred to as the lower VL as appropriate) of the lower layer encoder 25 shown in FIG.
C unit 36) and the lower layer GOV
Of the upper layer GOP so that the upper layer VOP displayed at the same time as or immediately after the display time of the first displayed VOP becomes the first displayed VOP of the upper layer GOV. Is the VLC unit 36 (hereinafter referred to as “upper V” as appropriate) of the upper layer encoder 23 shown in FIG.
LC unit 36).

The GOV configuration process performed by the lower VLC unit 36 or the upper VLC unit 36 will be described with reference to the flowchart of FIG. 43 or 44.

First, with reference to the flowchart of FIG. 43, the process of configuring the GOV by the lower VLC unit 36 will be described.

In the lower VLC unit 36, first, in step S1, it is determined whether or not the data of the VOP of the lower layer being processed is the first encoded data in the GOV. If it is determined that the data is not the first encoded one, the process returns to step S1 after waiting for the supply of the data of the next lower layer VOP. If it is determined in step S1 that the VOP data of the lower layer to be processed is the first encoded data in the GOV, the process proceeds to step S2, and the first encoded data is processed. A GOV header is inserted before the data of the VOP, and the process proceeds to step S3. In step S3, GO is assigned to the lower layer.
A GOV header insertion signal indicating that the V header has been inserted is output to the upper VLC unit 36 together with the display time of the first VOP displayed in the lower layer GOV, and data of the next lower layer VOP is supplied. Then, the process returns to step S1.

Next, a description will be given, with reference to the flowchart in FIG. 44, of the GOV configuration processing by the upper VLC unit.

In the upper VLC unit 36, first, in step S11, it is determined whether or not the GOV header insertion signal has been transmitted from the lower VLC unit 36. If it is determined that the GOV header insertion signal has not been transmitted, the process proceeds to step S11.
Return to When it is determined in step S11 that the GOV header insertion signal has been transmitted, the process proceeds to step S12, in which the first displayed VOP of the lower layer GOV transmitted with the GOV header insertion signal is transmitted. The display time is recognized. Further, step S
In 12, the VOP of the upper layer displayed at or immediately after the display time is the GOP of the upper layer.
Is inserted into the coded bitstream so that it is displayed first in step S1.
Return to 1.

In the decoder of FIG. 13, when random access is instructed, time_code and VOP placed in the GOV header in the encoded bit stream
Modulo_time_base and VOP_time_i located in header
As described above, the display time of the VOP to be randomly accessed is obtained using ncrement, and random access is performed based on the display time.

[0233] Next, in MPEG4, a sequence (picture structure) of a picture type of a VOP constituting the lower layer and the upper layer is not particularly specified.

In the encoder of FIG. 1, for example,
For realizing spatial scalability, a sequence of objects constituting an image to be encoded is, for example,
When hierarchized into two or more layers, such as a lower layer and an upper layer, the VOPs of the upper layer are encoded in the same order as the display order.

FIG. 45 shows the picture structure of the upper layer and the lower layer in the spatial scalable coding. In FIG. 45, the # i-th I-VOP, P-VOP, or B-VOP in the display order of the lower layer is represented as _Ibi , _Pbi , or _Bbi , respectively, and the upper layer is represented by _Ibi , _Pbi , or _Bbi . , # I-th P-
The VOP or B-VOP, respectively, P _ei or B _ei
Respectively.

[0236] In Figure 45, I _b3 of the lower layer is I-VOP is encoded / decoded first. Then, using the I _b3 of the lower layer as a reference picture, B _b1 are first displayed in the lower layer is encoded / decoded. Since the lower layer VOP displayed at the same time is basically used as the reference image for the VOP of the upper layer of spatial scalable coding, after encoding / decoding of B _b1 of the lower layer,
_{Be1 of the} upper layer using this as a reference image is encoded / decoded. Next, B _b2 displayed second in the lower layer is the I / O of the lower layer that has already been encoded / decoded.
_b3 encoded / decoded as a reference image, B _e2 which is an upper layer to be displayed on the same time, the B _b2 of the lower layer,
Encoding / decoding is performed using the upper layer _Be1 as a reference image.

Then, the upper layer _Pe3 is encoded / decoded using the lower layer _Ib3 as a reference image. Thereafter, P _b6 of the lower layer as a reference picture for encoding / decoding of the lower layer B _b4 is encoded / decoded, and P _b6 of the lower layer, again using I _b3 of the lower layer as a reference picture , The lower layer B _b4 is encoded / decoded.
Then, using the lower layer B _b4 and the upper layer P _e3 as reference images, the upper layer _{Be 4} is encoded / decoded.

Thereafter, B _b5 of the lower layer is encoded / decoded using I _b3 and P _b6 of the lower layer as reference images, and
Using the lower layer B _b5 and the upper layer _{Be 4} as reference images, the upper layer _{Be 5} is encoded / decoded. Then, P _b6 of lower layer, because it is already coded / decoded, a P _b6 of the lower layer, as a reference image B _e5 of the upper layer, B _e6 pertaining to the high-level layer is encoded / decoded.

Thereafter, the lower layer B_b7As a reference image
P of lower layer used_b9Is encoded / decoded,
P of lower layer of_b9And the lower layer P_b6See
As an image, B of lower layer_b7Are encoded / decoded.
Then, B of the lower layer _b7And B in the upper layer_e6To
As a reference image, B of the upper layer_e7Is encoded / decoded
And the P of the lower layer that has already been encoded / decoded_b6And
And P_b9Is used as a reference image, and B in the lower layer_b8Is encoded /
Decrypted. And B of the lower layer_b8And
Ear B_e7Is used as a reference image, and B in the upper layer_e8Is the sign
B_e8And the lower layer
P_b9Is used as a reference image, and B in the upper layer_e9Is encoding / decoding
No.

Therefore, if attention is paid to the lower layer, B_b1,
B_b2, I_b3, B_b4, B_b5, P_b6, B _b7, B_b8, P_b9Order
The VOP indicated by_b3, B_b1, B_b2, P_b6,
B_b4, B _b5, P_b9, B_b7, B_b8Are encoded / decoded in the order
To go.

On the other hand, when attention is paid to the upper layer, B
_e1, B_e2, P_e3, B_e4, B_e5, B_e6, B_e7, B_e8, B_e9
Is displayed in the order of_e1, B_e2,
P_e3, B _e4, B_e5, B_e6, B_e7, B_e8, B_e9Sign in the order
And decrypted. That is, the VOP of the upper layer is
Are encoded / decoded in the same order as the display order.

As described above, when the VOPs of the upper layer are encoded / decoded in the same order as the display order, the VOPs are encoded / decoded in a different order from the display order. In addition, the processing can be facilitated.

Here, even when spatial scalable encoding is performed, the encoder / decoder shown in FIG. 1 can perform encoding / decoding on a picture structure different from the picture structure shown in FIG. 45. Is possible. That is, the picture structure that causes the encoder in FIG. 1 / the decoder in FIG. 13 to perform encoding / decoding is not limited to that shown in FIG. Similarly,
The reference relation and the encoding / decoding order are not limited to those shown in FIG.

Note that the reference relationship shown in FIG.
This satisfies the reference relationship defined by ref_select_code of G4. In addition, when performing spatial scalable encoding, a method of encoding / decoding upper layer VOPs in the same order as the display order is applicable regardless of the presence or absence of GOV.

By the way, in the picture structure shown in FIG. 45, the P displayed in the third upper layer
_The VOPs displayed after _e3 are all B-VOPs.

[0246] On the other hand, the display time of the VOP corresponds to the modu of the VOP header.
Based on lo_time_base and VOP_time_increment, time_code of GOV header or modu of I / P-VOP displayed immediately before
It is calculated based on the synchronization point indicated by lo_time_base (however, modulo_time of the I / P-VOP displayed immediately before
Since the synchronization point indicated by _base, that is, the display time with the second precision of the I / P-VOP displayed immediately before can also be obtained based on the synchronization point indicated by time_code of the GOV header, The display time of the VOP is, after all, G
It can be said that it is obtained based on the synchronization point indicated by the time_code of the OV header).

Accordingly, for the lower layer in FIG.
The display time of B _b1 , B _b2 , I _b3 is the time_code of the GOV header
, The display time of B _b4 , B _b5 , and P _{b6 is} the display time of the second precision of I _b3 displayed immediately before it, and the display time of B _b7 , B _b8 , and P _b9 is displayed immediately before that The display time of the second precision of P _b6 ,
It is calculated using each.

On the other hand, for the upper layer in FIG.
_e1, the display time of B _e2, P _e3 is calculated using the time_code of the GOV header. Then, VOP to be displayed after P _e3 is,
As described above, since all are B-VOPs,
Display time of -VOP is calculated using the display time of one-second resolution of P _e3 displayed immediately before. As a result, the number of bits of the modulo_time_base of B-VOP is displayed is increased after P _e3, coding efficiency is degraded.

That is, the modulo_time of the noted VOP of interest
_base arranges 0 in the least significant bit, and in the upper bit, the second precision of the display time of the I / P-VOP displayed immediately before,
The same number of 1s as the difference between the display time of the target VOP and the second precision is arranged. Therefore, modulo_time_base is
If there is a P-VOP, it is reset to "0", so to speak, thereby preventing an increase in the number of bits.

However, for the upper layer shown in FIG. 45, all VOPs displayed after _Pe3 are B-
Since this is a VOP, the modulo_time_base is not reset, and the number of bits increases.

[0251] Specifically, as shown in FIG. 46, the lower layer, after each _{I b3, P b6, P b6} , modulo_time_base is reset to "0". Then, the B _b7 or B _b10 after P _b6, after B _b4, P _b6 after I _b3 is, I _b3, P _b6, from P _b6, B _b4,
The modulo_tim in which the number “1” corresponding to the second precision of each display interval up to B _b7 and B _b10 is reset to “0”
It is added to the upper bits of e_base (in FIG. 46, I _b3 , P
_Since the display intervals from _b6 and _Pb6 to _Bb4 , _Bb7 and _Bb10 are all less than 1 second, the respective mo
dulo_time_base is the number of “0” as upper bits of “0”.
"0" to which "1" is added).

On the other hand, for the upper layer, P
In After _e3, after modulo_time_base is reset to "0", S S display time accuracy of the display time as a reference accuracy P _e3 are each increase of 1 sec, subjected to VOP m
odulo_time_base also increases by one bit. As a result, as shown in FIG. 46, after P _e3 , modulo_t
After ime_base is reset to "0", B _e4, the subsequently displayed _{B e5, B e6, B e7} , B e8, mod of B _e9, B _e10
ulo_time_base is "0", "10", "10", "110", "11
0 "," 1110 "," 1110 ", and the number of bits increases.

Therefore, for example, as shown in FIG.
In the spatial scalable coding, when the encoding / decoding of the upper layer is performed in the same order as the display order, the modulo_time_base is set to "" even after the B-VOP of the upper layer.
0 ”, and the same number of“ 1 ”as the difference between the display time of the B-VOP and the next displayed VOP with the second precision,
VOP to be displayed next by adding it as upper bits of 0 "
Make up modulo_time_base. in this case,
The display time of the VOP of the upper layer is calculated based on the display time of the second precision of the VOP (which may be any one of I, P, and B) displayed immediately before the VOP. .
If there is no VOP displayed before the VOP of interest,
That is, the target VOP is the first V displayed in the GOV.
In the case of the OP, since there is no VOP displayed on the straight line, for the first displayed VOP, for example, the configuration of the modulo_time_base and the calculation of the display time are calculated based on the time_code. .

FIG. 47 shows the modulo_time_base of the picture structure shown in FIG. 45 when the method of configuring modulo_time_base and the method of calculating the display time of the VOP are changed as described above.

In this case, the upper layer is displayed first.
For B _e1 , the second accuracy of the display time (0 in FIG. 47)
h: 12m: 34s) and time_code (0h: 12m: 34s in FIG. 47)
Is 0, so the modulo_time_base is "
"A, 0 for" 0 for even .B _e2 becomes "obtained by adding" 1 0 ", the second resolution (Figure 47 of the display time, 0h: 12
m: a 34s), the second resolution (Figure 47 the display time of B _e1 displayed immediately before, 0h: 12m: from the difference between the 34s) is 0, the modulo_time_base is, "0", 0 Pieces of "1"
Is added to “0”. Regarding _Pe3 , the second precision of the display time (0h: 12m: 35s in FIG. 47) and the second precision of the display time of _Be2 displayed immediately before (0h: 1 in FIG. 47).
2m: 34s) is 1, so the modulo_time_ba
se is “10” obtained by adding one “1” to “0”. For _Be4 , the second accuracy of the display time (0h: 12m: 35s in FIG. 47) and the second accuracy of the display time of _Pe3 displayed immediately before (0h: 12m: 35s in FIG. 47) Is 0
Therefore, the modulo_time_base becomes “0” obtained by adding zero “1” to “0”. That is, since the difference between the seconds accuracy of the display time of B _e4, the second resolution of the display time of P _e3 displayed immediately before is less than 1 second, so to speak, in the B _e4, modulo_time_base is reset.

[0256] For B _e5 is (in FIG. 47, 0h: 12m: 36s) seconds accuracy of the display time and, in second resolution (Figure 47 the display time of B _e4 displayed immediately before, 0h: 12m: Since the difference from 35s) is 1, its modulo_time_base is "
It becomes “0” which is obtained by adding one “1” to “0.” For _Be6 , the second precision of the display time (0h: 12 in FIG. 47)
m: a 36 s), the second resolution (Figure 47 the display time of B _e5 to be displayed immediately before, 0h: 12m: from the difference between 36 s) is 0, the modulo_time_base is, similarly to the case of B _e4 Is reset to "0".

[0257] In the same _{manner, B e7, B e8, B} e9, mo of B _e10
dulo_time_base is “1” as shown in FIG.
0 "," 0 "," 10 "," 0 ".

Next, the process of setting the modulo_time_base (modulo_time_base setting process) is performed for the lower layer or the upper layer by using the VLC unit 36 (lower VLC unit 36) of the lower layer encoding unit 25 shown in FIG. VLC unit 3 of the upper layer encoding unit 23 shown in FIG.
6 (upper VLC unit 36).

The following describes the modulo_time_base setting processing performed by the upper VLC unit 36 with reference to the flowchart shown in FIG.

In the upper VLC unit 36, the VOP currently being processed is set as the target VOP, and in step S21, a difference D between the target VOP and the VOP displayed immediately before is displayed with a second-accuracy display time. Proceeds to step S22,
It is determined whether D is equal to zero. Step S22
When it is determined that D is equal to 0, the process proceeds to step S23, modulo_time_base is reset to "0", and the process proceeds to step S28. In step S28, "
The modulo_time_base reset to "0" is set as the target VOP, and the process returns to step S21 after waiting for the supply of the next VOP to be processed.

On the other hand, if it is determined in step S22 that D is not equal to 0, the process proceeds to step S24,
modulo_time_base is reset to “0”. And
In step S25, “1” is added as the most significant bit of modulo_time_base, and the process proceeds to step S26. In step S26, D is decremented by 1, and the process proceeds to step S27 to determine whether D is equal to 0. If it is determined in step S27 that D is not equal to 0, the process returns to step S25, and
Until it is determined in step S27 that D is equal to 0, the processing of steps S25 to S27 is repeated.

Then, in step S27, D is set to 0.
Is determined, that is, modulo_time_base in which the same number of “1” as the difference between the display time of the second precision and the noticed VOP and the VOP displayed immediately before are added to “0” is formed. If so, the process proceeds to step S28, and the modulo_tim
e_base is set to the VOP of interest and the VOP to be processed next
, And returns to step S21.

In the lower VLC unit 36, in step S21 in FIG. 48, the target VOP and the V
The difference D between the target VOP and the immediately preceding I / P-VOP displayed in second precision, not the difference in display time in second precision from the OP (I / P / B-VOP), is obtained. Except for the top VL
A modulo_time_base setting process similar to that of the C unit 36 is performed.

In the decoder of FIG. 13, the display time of the VOP of the lower layer is obtained with reference to the display time (or time_code) of the second precision of the I / P-VOP displayed immediately before, but the display time of the upper layer is The display time of the VOP is determined based on the display time (or time_code) with the second precision of the VOP (regardless of I, P, B) displayed immediately before.

[0265] Incidentally, in the case shown in FIG. 45, the coding / decoding of B _e1 pertaining to the high-level layer is carried out only B _b1 of the lower layer to be displayed at the same time as a reference picture.

The upper layer B _e1 is a B-VOP, but the BV
OP is normally to be encoded so that fewer bits generated than I / P-VOP, the image quality of the decoded image of B _e1 is likely to deteriorate. Furthermore, if the quality of the decoded image of B _e1 deteriorates is likely to deteriorate image quality of the decoded image of B _e2 using the B _e1 as a reference picture. That is, FIG.
In the case shown in (2), the image quality of the B-VOP displayed before the P-VOP _Pe3 is displayed in the upper layer is likely to deteriorate.

In order to prevent (reduce) such image quality deterioration, for example, for a picture structure for realizing spatial scalability shown in FIG. , FIG.
As shown in FIG.

That is, in FIG. 49, when the I-VOP of the lower layer is encoded / decoded, the P-VOP displayed in the upper layer at the same time as the display time of the I-VOP of the lower layer
Are encoded / decoded immediately after the lower layer I-VOP.

Therefore, in the case shown in FIG. 49,
First, I _b3 is I-VOP which is first displayed in the lower layer is encoded / decoded. And I _{b3 of} the lower layer
P _e3 which is the P-VOP of the upper layer displayed at the same time as
Is encoded / decoded using encoded / decoded I _{b 3} as a reference image immediately before.

[0270] Then, further, using I _b3 of the lower layer as a reference picture, B _b1 of the lower layer is encoded / decoded. Next, B _e1 pertaining to the high-level layer is encoded / decoded, the coding / decoding of B _e1 of the upper layer, as well as B _b1 of the lower layer, P _e1 of the upper layer also as a reference image Used. This is because, in MPEG4, for B-VOP in spatial scalable coding, forward prediction (forward
prediction), "most recent decoded enhance
ment VOP of the same layer ”is used as a reference image. In FIG. 49,“ most recent decoded enhancement VOP of the same layer ”is used.
ame layer "is because it is P _e3.

After the encoding / decoding of the upper layer B _e1 , the lower layer B _b2 is encoded / decoded similarly using the lower layer I _b3 as a reference image, and thereafter, the upper layer B _e1 and the lower layer B _b1 are encoded / decoded. with B _b2 as a reference picture, B _e2 of the high-level layer is encoded / decoded. Subsequently, P _b6 of the lower layer is encoded / decoded, below, in the same manner as in FIG. 45, the lower layer B _b4, the upper layer B _e4,
B _{b5 of} lower layer, B _e5 of upper layer,
P _b9, the lower layer B _b7, the upper layer B _e7, the lower layer B _b8, is encoded / decoded in order of the upper layer B _e8, B _e9.

As described above, when the I-VOP of the lower layer is encoded / decoded, the P-VOP displayed at the same time as the display time of the I-VOP of the lower layer is changed to the lower layer in the upper layer. By encoding / decoding immediately after the I-VOP of the layer, in the upper layer, the B-VOP displayed first is encoded / decoded using the P-VOP as a reference image. (In FIG. 49, P _e3 is used as a reference image,
Since _Be1 is encoded / decoded), it is possible to prevent the image quality of the B-VOP displayed before the P-VOP from deteriorating in the upper layer.

Here, the encoder / decoder shown in FIG. 1 performs encoding / decoding on the basis of the reference relationship other than the reference relationship and the encoding / decoding order shown in FIG. 49 and the encoding / decoding order. It is possible to do. That is, the reference relation and the encoding / decoding order are not limited to those shown in FIG.

Note that the reference relationship shown in FIG.
This satisfies the reference relationship defined by ref_select_code of G4. In addition, when performing spatial scalable encoding, a method of performing encoding / decoding in the order shown in FIG.
Applicable with or without GOV.

Further, when encoding / decoding is performed according to the reference relationship shown in FIG. 49, the encoding / decoding order may be different from the above-described case. That is, for example, it is possible to encode / decode all VOPs in the lower layer and then encode / decode the VOPs in the upper layer. However, in FIG. 49, since the lower layer VOP is used as the reference image for encoding / decoding the upper layer VOP, in this case, the reference image is used as the reference image for encoding / decoding the upper layer VOP. The VOP of the lower layer to be used is the VOP of the upper layer
May need to be held for a long time until the encoding / decoding of is completed. In order to avoid such a long holding, it is desirable to adopt the encoding / decoding order as described with reference to FIG. This is shown in FIG.
The same applies to the case shown in FIG.

Next, in the present embodiment, in order to prevent quick and efficient random access from being hindered, as shown in FIG. 39 to FIG. 42, the GOV of the lower layer is displayed first. The VOP of the upper layer displayed at the same time as or immediately after the display time of the upper VOP is displayed first at the top of the GOV of the upper layer.
The upper layer is divided into GOVs.

However, in the time_code of the GOV header, since the display time of the VOP displayed first in the GOV is set with the second precision, the display time of the VOP displayed first in the lower layer GOV is set. When the GOV of the upper layer is configured so that the VOP of the upper layer displayed immediately after is displayed first in the GOV of the upper layer, the value set in the time_code of the GOV of the lower layer ( Second precision first display time) and GO of the lower layer
The value set in the time_code of the GOV of the upper layer corresponding to V may not match.

That is, FIG. 50 shows a configuration example of a lower layer and an upper layer when performing temporal scalable coding.

Now, in FIG. 50, B _b2 of the lower layer
And a VOP which is first displayed, when implementing an GOV, for the upper layer is the VOP of the upper layer that appears immediately after the display time of B _b2 to be displayed first in the GOV subordinate layer B _e3 GOP of the upper layer is configured such that is the VOP displayed first. in this case,
Time_code of GOV header of GOV composed of lower layers
In, the display time of B _b2 displayed first in the GOV is set with second precision. Similarly, the time_code of the GOV header of GOV composed of an upper layer, the first display time of B _e3 displayed in the GOV is set in seconds accuracy.

That is, in FIG. 50, the display time of B _b2 is 01
h: 12m: 01h from the 33s: 12m: is the time between the 34s, the display time of B _e3 is, 01h: 12m: 01h from the 34s: 12m: because it is the time between the 35s, composed of a lower layer T of GOV header of GOV
01h: 12m: 33s is set in ime_code, and 01h: 12 is set in time_code of the GOV header of the GOV composed of the upper layer.
m: 34s is set. Therefore, the tim of the lower layer GOV
The value set in e_code does not match the value set in time_code of the GOV of the upper layer corresponding to the GOV of the lower layer.

As described above, the GOV of the lower layer is
Set to the time_code of the
If the value set in the time_code of the GOV of the upper layer corresponding to V does not match, when random access is made to the encoded bit stream, the lower layer and the upper layer have different second-accurate times (time_code Based on the indicated synchronization point), the display time of the VOP must be calculated.

Therefore, the time_code of the GOV of the upper layer
Contains the GOV of the lower layer corresponding to the upper layer.
The same value as the value set in time_code can be set.

That is, FIG. 51 shows an example of the configuration of the lower layer and the upper layer when performing temporal scalable encoding similar to the case shown in FIG.

Now, in FIG. 51, as in the case of FIG. 50, B _b2 of the lower layer is changed to the VO displayed first.
As When the P, configuring the GOV, the upper layer is the VOP of the upper layer that appears immediately after the display time of B _b2 to be displayed in the first GOV of the lower layer B _e3
Is the first VOP displayed, so that G in the upper layer
The OP is configured.

[0285] Then, in this case, the GOV header of the GOV composed of a lower layer time_code, as in FIG. 50, a first display time of B _b2 to be displayed in the GOV seconds accuracy Set. Therefore, the time_code of the GOV header of the GOV composed of the lower layer is 01h: 12m: 33
s is set.

On the other hand, GOV GO composed of upper layers
In the time_code of the V header, the same value as the value set in the time_code of the GOV of the lower layer corresponding to the GOV of the upper layer is set. Therefore, 01h: 12m: 33s is also set in the time_code of the GOV header of the GOV formed in the lower layer.

In this case, when performing random access to the encoded bit stream, the display time of the VOP must be calculated in the lower layer and the upper layer based on different second-accurate times (synchronization points indicated by time_code). Can be prevented. That is, in the lower layer and the upper layer, V is always based on the same second-accurate time (synchronization point indicated by time_code).
The display time of the OP can be calculated.

[0288] The GOV GO composed of upper layers
Even when the same value as the time_code of the GOV of the lower layer corresponding to the GOV of the upper layer is set in the time_code of the V header, the G configured by the upper layer
In the encoder of FIG. 1, modulo_time_base and VOP_time_increment can be obtained in the same manner as in the case where the display time of the first VOP displayed in the GOV is set to the time_code of the GOV header of the OV with the second precision. In the decoder of FIG. 13, the modulo_time_base and VOP_
The display time of the VOP can be obtained by using time_increment and time_code.

Next, the encoder and decoder described above can be realized by dedicated hardware, or can be realized by causing a computer to execute a program for performing the above-described processing. Can be.

That is, FIG. 52 shows an example of the configuration of an embodiment of a computer functioning as the encoder in FIG. 1 or the decoder in FIG.

The ROM (Read Only Memory) 201 stores, for example, a boot program. CPU
(Central Processing Unit) 202 is, for example, an HD
(HardDisk) The program stored in 206 is stored in RAM
(Read Only Memory) 203 is loaded and executed to perform various processes. RAM 20
3 is a program executed by the CPU 202 and the CPU 2
02 is temporarily stored. The input unit 204 includes, for example, a keyboard and a mouse, and is operated to input necessary commands and data. The output unit 205 includes, for example, a display and displays data according to the control of the CPU 202. The HD 206 stores a program to be executed by the CPU 202, as well as image data to be encoded, encoded data (encoded bit stream), decoded image data, and the like. The communication I / F (Interface) 207 controls communication with the outside, for example, receives image data to be encoded from the outside, and transmits an encoded bit stream after encoding to the outside. It has been made to be. Further, the communication I / F 207 is configured to receive an externally encoded bit stream, and to transmit decoded image data to the outside.

The computer C configured as described above
By causing the PU 202 to execute a program for performing the above-described processing, the computer functions as the encoder illustrated in FIG. 1 and the decoder illustrated in FIG.

The scope of the present invention is not limited to the range based on MPEG4.

[0294]

According to the present invention, an image encoding apparatus and an image encoding apparatus are provided.
According to the conversion method , the sequence of objects constituting an image is
The first hierarchy is a lower hierarchy, and the first hierarchy is a higher hierarchy.
Layered into two levels, the first
The first or second level is divided into one or more groups
Each is encoded. Further, the first or second hierarchy
Table of objects initially displayed in a group
The second display time, which is the second
Each of the objects in the first or second hierarchy
To the object based on the second display time
Seconds relative time information indicating the seconds accuracy of the display time of the event is added.
Is done. Then, the second-precision phase is added to the object of the second hierarchy.
When time-to-time information is added, adjacent objects in display order
Based on the display time difference between
The relative second time information added to the object is reset.
Is Therefore, it is possible to prevent the coding efficiency from deteriorating.

According to the image decoding apparatus and the image decoding method of the present invention , the sequence of objects constituting an image is
A first hierarchy which is a lower hierarchy and a second hierarchy which is an upper hierarchy
Hierarchized into layers, the first or
Divide the second layer into one or more groups and code each
Into a first or second level group,
Seconds precision that indicates the display time of the displayed object with second precision
First or second level, including the first display time
Based on the first display time in seconds for each object
, The second representing the second precision of the display time of the object
Adds precision relative time information to the second layer object
When adding the second precision relative time information to the
Based on the difference between the display times of the objects
Second relative time information to be added to the object at
Coded bitstream obtained by reset
The system is received and decoded . Therefore, it is possible to decode the coded bit stream in which the deterioration of the coding efficiency is prevented.

According to the recording medium of the present invention , an image
The sequence of objects that are
Layer and a second layer, which is a higher layer,
The first or second layer of the sequence of events
Each group is encoded and the first or second
The first visible object in a group in the hierarchy
The first display time of second precision, which represents the display time of
Including the objects of the first or second hierarchy
To the object based on the second display time
Seconds relative time information indicating the second accuracy of the display time of the event is added.
And second-second relative time information is stored in the second layer object.
When adding, objects that are adjacent in the display order
Based on the display time difference, the objects in the second layer
By resetting the second precision relative time information to be added
The resulting coded bit stream is recorded. Therefore, it is possible to provide an encoded bit stream in which the encoding efficiency is prevented from deteriorating.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an encoder to which the present invention has been applied.

FIG. 2 is a diagram for explaining that a position and a size of a VO change with time.

3 is a block diagram showing a configuration example of VOP encoding unit 3 ₁ to 3 _N in FIG.

FIG. 4 is a diagram for explaining spatial scalability.

FIG. 5 is a diagram for explaining spatial scalability.

FIG. 6 is a diagram for describing spatial scalability.

FIG. 7 is a diagram for describing spatial scalability.

FIG. 8 is a diagram for explaining a method of determining size data and offset data of a VOP.

9 is a block diagram illustrating a configuration example of a lower layer encoding unit 25 in FIG.

10 is a block diagram illustrating a configuration example of an upper layer encoding unit 23 in FIG.

FIG. 11 is a diagram for describing spatial scalability.

FIG. 12 is a diagram for explaining time scalability.

FIG. 13 is a block diagram illustrating a configuration example of an embodiment of a decoder to which the present invention has been applied.

14 is a block diagram illustrating another configuration example of the VOP decoding unit 72 ₁ to 72 _N of Figure 13.

15 is a block diagram illustrating a configuration example of a lower layer decoding unit 95 in FIG.

16 is a block diagram illustrating a configuration example of an upper layer decoding unit 93 in FIG.

[Fig. 17] Video Verification Model (Vers
FIG. 21 is a diagram illustrating the configuration of an encoded bit stream in (Ion10.0).

FIG. 18 is a diagram illustrating the syntax of a VS.

FIG. 19 is a diagram illustrating the syntax of a VO.

FIG. 20 is a diagram illustrating the syntax of a VOL.

FIG. 21 is a diagram illustrating the syntax of a VOL.

FIG. 22 is a diagram illustrating the syntax of a VOL.

FIG. 23 is a diagram illustrating the syntax of a VOL.

FIG. 24 is a diagram illustrating the syntax of a VOL.

FIG. 25 is a diagram illustrating the syntax of a VOL.

FIG. 26 is a diagram illustrating the syntax of GOV.

FIG. 27 is a diagram illustrating the syntax of GOV.

FIG. 28 is a diagram illustrating the syntax of a VOP.

FIG. 29 is a diagram illustrating the syntax of a VOP.

FIG. 30 is a diagram illustrating the syntax of a VOP.

FIG. 31 is a diagram illustrating the syntax of a VOP.

FIG. 32 is a diagram illustrating the syntax of a VOP.

FIG. 33 is a diagram illustrating the syntax of a VOP.

FIG. 34 is a diagram illustrating VOP syntax.

FIG. 35 is a diagram illustrating the syntax of a VOP.

FIG. 36 is a diagram illustrating the syntax of a VOP.

[FIG. 37] time_code, modulo_time_base, VOP_time_in
It is a figure which shows the relationship of crement.

FIG. 38: time_code, modulo_time_base, VOP_time_in
It is a figure which shows the relationship of crement.

FIG. 39 is a diagram illustrating a state in which GOVs are independently configured in a lower layer and an upper layer.

FIG. 40 is a diagram showing a state in which a lower layer and an upper layer independently constitute a GOV.

FIG. 41 shows the GOV of the upper layer as the GOV of the lower layer.
It is a figure showing the state constituted in association with.

FIG. 42 shows the GOV of the upper layer as the GOV of the lower layer.
It is a figure showing the state constituted in association with.

FIG. 43 shows the GOV of the upper layer as the GOV of the lower layer.
10 is a flowchart for explaining the processing of the VLC unit 36 in FIG.

FIG. 44 shows the GOV of the upper layer as the GOV of the lower layer.
11 is a flowchart for explaining the processing of the VLC unit 36 in FIG.

FIG. 45 is a diagram illustrating an example of a picture structure of an upper layer and a lower layer in spatial scalable encoding.

[Fig. 46] Fig. 46 is a diagram for describing a case where the number of bits of modulo_time_base increases.

[Fig. 47] Fig. 47 is a diagram for describing a method for preventing an increase in the number of bits of modulo_time_base.

[Fig. 48] Fig. 48 is a diagram for describing processing for preventing an increase in the number of bits of modulo_time_base.

[Fig. 49] Fig. 49 is a diagram for describing an encoding / decoding order and a reference relationship for preventing image quality deterioration.

[Fig. 50] Fig. 50 is a diagram illustrating a state where the time_codes of the lower layer and the upper layer do not match.

[Fig. 51] Fig. 51 is a diagram for describing a method of matching time_codes of a lower layer and an upper layer.

FIG. 52 is a block diagram illustrating a configuration example of another embodiment of an encoder and a decoder to which the present invention has been applied.

FIG. 53 is a block diagram illustrating a configuration of an example of a conventional encoder.

FIG. 54 is a block diagram illustrating a configuration of an example of a conventional decoder.

[Explanation of symbols]

1 VO component, 2 _{1 to} 2 _N VOP component, 3 ₁
To 3 _N VOP encoding unit, 4 multiplexing unit, 21
Image layering unit, 23 upper layer encoding unit, 24 resolution conversion unit, 25 lower layer encoding unit, 26 multiplexing unit, 31 frame memory, 32 motion vector detector, 33 arithmetic unit, 34 DCT unit, 35
Quantizer, 36 VLC unit, 38 inverse quantizer,
39 IDCT unit, 40 arithmetic unit, 41 frame memory, 42 motion compensator, 53 frame memory,
71 demultiplexer, 72 _{1 to} 72 _N VOP decoder,
73 image reconstruction unit, 91 demultiplexing unit, 93
Upper layer decoding unit, 94 resolution conversion unit, 95 lower layer decoding unit, 102 IVLC unit, 103 inverse quantizer, 104 IDCT unit, 105 arithmetic unit, 1
06 frame memory, 107 motion compensator, 11
2 frame memory, 201 ROM, 202 CP
U, 203 RAM, 204 input unit, 205 output unit, 206 HD, 207 communication I / F

────────────────────────────────────────────────── ─── Continuation of front page (56) References JP-A-8-98143 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N ⁷ /24-7/68 H04N 5 / 91-5/95

Claims (5)

(57) [Claims] [Claim 1] An image was encoded, an image encoding apparatus that outputs a coded bit stream obtained as a result of the lower floor of the sequence of objects constituting the image
Layering means for layering into a first layer, which is a layer, and a second layer, which is a higher layer; and at least one first or second layer of the sequence of objects output by the layering means. Encoding means that respectively encodes the first and second hierarchical groups, and includes a second-precision leading display time that represents the display time of the first displayed object in second precision in the first or second layer group; For each of the objects in the first or second hierarchy,
Referenced to the second resolution first display time comprises adding means for adding second resolution relative time information representing a second resolution of the display time of the object, said additional means, said second object of the second hierarchy Add precision relative time information
At this time, based on the display time difference between the objects adjacent in the display order, the object is added to the object in the second hierarchy.
Image coding apparatus characterized by resetting the second resolution relative time information. 2. An image encoding, an image encoding method which outputs a coded bit stream obtained as a result of the lower floor of the sequence of objects constituting the image
A layering step of layering a first layer which is a layer and a second layer which is a higher layer ; and encoding the first or second layer of the sequence of objects into one or more groups, respectively. the first or second group of layers, the coding step that including each second resolution first display time representing a display time of an object to be the first in the second resolution, the first or second hierarchical For each object of
An adding step of adding second-accuracy relative time information indicating the second accuracy of the display time of the object with reference to the second-accuracy leading display time , wherein in the adding step, the object of the second hierarchy is added.
Resetting the second-precision relative time information to be added to the object in the second hierarchy based on a difference in display time between objects adjacent in the display order when adding the second-precision relative time information to the object. An image encoding method characterized by the following. 3. An image decoding apparatus for decoding an image, the lower floor a sequence of objects composing the picture
Hierarchized into a first layer that is a layer and a second layer that is a higher layer, the first or second layer of the sequence of objects is divided into one or more groups and encoded, and the first or second layer is encoded. The second hierarchy group includes a second-precision head display time that represents the display time of the first displayed object in second precision, and the first or second hierarchy object includes:
Adds second-precision relative time information that represents the second precision of the display time of the object, based on the second-precision first display time
Then, the second-precision relative time information is added to the object of the second hierarchy.
When adding information, the
Based on the difference between the display times of the objects,
Resets the relative second time information added to the project.
Image decoding apparatus characterized by comprising: a receiving means for receiving a coded bit stream obtained, and decoding means for decoding the coded bit stream by that. 4. An image decoding method for decoding an image, the lower floor a sequence of objects composing the picture
Hierarchized into a first layer that is a layer and a second layer that is a higher layer, the first or second layer of the sequence of objects is divided into one or more groups and encoded, and the first or second layer is encoded. The second hierarchy group includes a second-precision head display time that represents the display time of the first displayed object in second precision, and the first or second hierarchy object includes:
Adds second-precision relative time information that represents the second precision of the display time of the object, based on the second-precision first display time
Then, the second-precision relative time information is added to the object of the second hierarchy.
When adding information, the
Based on the difference between the display times of the objects,
Resets the relative second time information added to the project.
That receives the encoded bit stream obtained by the image decoding method characterized by decoding the encoded bit stream. 5. A recording medium on which an encoded bit stream obtained by encoding an image is recorded, wherein a sequence of objects forming the image is stored in a lower order floor.
Hierarchized into a first layer that is a layer and a second layer that is a higher layer, the first or second layer of the sequence of objects is divided into one or more groups and encoded, and the first or second layer is encoded. The second hierarchy group includes a second-precision head display time that represents the display time of the first displayed object in second precision, and the first or second hierarchy object includes:
Adds second-precision relative time information that represents the second precision of the display time of the object, based on the second-precision first display time
Then, the second-precision relative time information is added to the object of the second hierarchy.
When adding information, the
Based on the difference between the display times of the objects,
Resets the relative second time information added to the project.
Recording medium obtained coded bit stream is characterized in that it is recorded by that.