JP3344577B2 - Image encoding device and image encoding method, image decoding device and image decoding method, and recording method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image encoding device and an image encoding method, an image decoding device and an image decoding method, and a recording medium and a recording method. In particular,
For example, moving image data is recorded on a recording medium such as a magneto-optical disk or a magnetic tape, and is reproduced and displayed on a display or the like, a video conference system, a video telephone system, a broadcasting device, a multimedia database search system. For example, when moving image data is transmitted from a transmitting side to a receiving side via a transmission path, and the receiving side receives and displays the moving image data, or edits and records the image, a suitable image is used. The present invention relates to an encoding device and an image encoding method, an image decoding device and an image decoding method, and a recording medium and a recording method.

[0002]

2. Description of the Related Art For example, in a system for transmitting moving image data to a remote place, such as a video conference system or a video telephone system, image data is converted into a line correlation or a frame in order to use a transmission path efficiently. The compression encoding is performed using the inter-correlation.

A moving picture experts group (MPEG) is a typical moving picture coding scheme.
(Moving picture coding for storage). This is ISO-I
Discussed in EC / JTC1 / SC2 / WG11,
It has been proposed as a standard, and employs a hybrid method combining motion compensation prediction coding and DCT (Discrete Cosine Transform) coding.

[0004] In MPEG, several profiles and levels are defined in order to support various applications and functions. The most basic is the main profile main level (MP @ ML (Main Profile
at Main Level)).

[0005] FIG. 42 is a diagram showing MP @ M in the MPEG system.
5 shows an exemplary configuration of an L encoder.

[0006] Image data to be encoded is input to a frame memory 31 and is temporarily stored. Then, the motion vector detector 32 reads out the image data stored in the frame memory 31 in units of macroblocks composed of, for example, 16 pixels × 16 pixels, and detects the motion vector.

Here, the motion vector detector 32 processes the image data of each frame as any one of an I picture, a P picture, and a B picture. It should be noted that it is determined in advance as to which of the I, P, and B pictures the image of each frame input sequentially is processed (for example, I, B,
P, B, P,..., B, P).

That is, the motion vector detector 32 refers to a predetermined reference frame in an image stored in the frame memory 31, and the reference frame and the current frame are to be encoded. 16 pixels of frame x 16
By performing pattern matching (block matching) with a small block (macro block) of the line, a motion vector of the macro block is detected.

[0009] Here, in MPEG, the image prediction modes include four modes: intra coding (intra-frame coding), forward prediction coding, backward prediction coding, and bidirectional prediction coding.
There are types, I pictures are intra-coded, P pictures are intra-coded or forward predicted coded, B pictures are intra-coded, forward predicted coded, backward predicted coded, or both methods predictive coded .

That is, the motion vector detector 32 sets the intra coding mode as the prediction mode for the I picture. In this case, the motion vector detector 32
The motion vector is not detected, and the prediction mode (intra prediction mode) is output to the VLC (variable length coding) unit 36 and the motion compensator 42.

The motion vector detector 32 performs forward prediction on a P picture and detects the motion vector. Further, the motion vector detector 32 compares a prediction error caused by performing forward prediction with, for example, a variance of a coding-target macroblock (a macroblock of a P picture), and the variance of the macroblock is more predictive. When the difference is smaller than the error, the intra coding mode is set as the prediction mode, and the prediction mode is output to the VLC unit 36 and the motion compensator 42. If the prediction error caused by performing the forward prediction is smaller, the motion vector detector 32 sets the forward prediction encoding mode as the prediction mode, and sets the VLC unit 36 and the motion compensator 42 together with the detected motion vector. Output to

Further, the motion vector detector 32 performs forward prediction, backward prediction, and bidirectional prediction on the B picture, and detects respective motion vectors. Then, the motion vector detector 32 performs forward prediction, backward prediction,
And a minimum prediction error of the bidirectional prediction (hereinafter, appropriately referred to as a minimum prediction error), and the minimum prediction error and the variance of the encoding target macroblock (the macroblock of the B picture), for example. Compare with As a result of the comparison, if the variance of the macroblock is smaller than the minimum prediction error, the motion vector detector 32 sets the intra coding mode as the prediction mode, and sets the VLC unit 3
6 and the motion compensator 42. If the minimum prediction error is smaller, the motion vector detector 32 sets the prediction mode in which the minimum prediction error is obtained as the prediction mode, and sets the VLC unit 3 together with the corresponding motion vector.
6 and the motion compensator 42.

The motion compensator 42 includes a motion vector detector 3
When both the prediction mode and the motion vector are received from 2,
According to the prediction mode and the motion vector, the coded and already locally decoded image data stored in the frame memory 41 is read out and supplied to the calculators 33 and 40 as a predicted image.

The arithmetic unit 33 reads from the frame memory 31 the same macroblock as the image data read from the frame memory 31 by the motion vector detector 32,
The difference between the macro block and the predicted image from the motion compensator 42 is calculated. This difference value is supplied to the DCT unit 34.

On the other hand, when only the prediction mode is received from the motion vector detector 32, that is, when the prediction mode is the intra-coding mode, the motion compensator 42 does not output a predicted image. In this case, the operator 33 (the operator 40
Does not perform any particular processing, and converts the macroblock read from the frame memory 31 into the DCT unit 34 without any processing.
Output to

In the DCT unit 34, the output of the arithmetic unit 33 is subjected to DCT processing, and the resulting DCT coefficient is supplied to the quantizer 35. In the quantizer 35,
A quantization step (quantization scale) is set corresponding to the amount of data stored in the buffer 37 (the amount of data stored in the buffer 37) (buffer feedback), and the DCT from the DCT unit 34 is set in the quantization step. The coefficients are quantized. The quantized DCT coefficients (hereinafter, appropriately referred to as quantization coefficients) are supplied to the VLC unit 36 together with the set quantization steps.

In the VLC unit 36, corresponding to the quantization step supplied from the quantizer 35,
The supplied quantization coefficient is converted into a variable-length code such as a Huffman code, for example, and output to the buffer 37.
Further, the VLC unit 36 performs a quantization step from the quantizer 35, a prediction mode (intra-coding (intra-picture prediction coding), a forward prediction coding,
A mode indicating which of the backward prediction coding and the bidirectional prediction coding has been set) and the motion vector are also variable-length coded and output to the buffer 37.

The buffer 37 temporarily stores data from the VLC unit 36 and smoothes the data amount.
Output to a transmission path or record on a recording medium.

The buffer 37 outputs the data storage amount to the quantizer 35, and the quantizer 35 sets a quantization step according to the data storage amount from the buffer 37. That is, when the buffer 37 is about to overflow, the quantizer 35 increases the quantization step, thereby reducing the data amount of the quantization coefficient. When the buffer 37 is about to underflow, the quantizer 35 reduces the quantization step, thereby increasing the data amount of the quantization coefficient. Thus, the overflow and the underflow of the buffer 37 are prevented.

The quantization coefficient and the quantization step output from the quantizer 35 are determined not only by the VLC unit 36 but also by the inverse quantizer 3.
8 as well. Inverse quantizer 35
Then, the quantized coefficient from the quantizer 35 is inversely quantized in accordance with a quantization step from the quantizer 35, and is thereby converted into a DCT coefficient. The DCT coefficient is supplied to an IDCT unit (inverse DCT unit) 39. I
In the DCT unit 39, the DCT coefficient is subjected to an inverse DCT process, and is supplied to the arithmetic unit 40.

The arithmetic unit 40 receives the output of the IDCT unit 39 and, as described above, the motion compensator 42 and the arithmetic unit 33.
Are supplied with the same data as the prediction image supplied to the calculation unit 40. The arithmetic unit 40 adds the signal (prediction residual) from the IDCT unit 39 and the prediction image from the motion compensator 42, The original image is locally decoded (however, when the prediction mode is intra coding, the output of the IDCT unit 39 is supplied to the frame memory 41 through the arithmetic unit 40). This decoded image is the same as the decoded image obtained on the receiving side.

The decoded image (local decoded image) obtained by the arithmetic unit 40 is supplied to and stored in the frame memory 41, and then inter-coded (forward predictive coding, backward predictive coding, quantitative predictive coding). ) Is used as a reference image (reference frame) for the image to be processed.

FIG. 43 shows an example of the configuration of an MPEG @ ML decoder in MPEG for decoding encoded data output from the encoder shown in FIG.

The encoded data transmitted via the transmission path is received by a receiving device (not shown), or the encoded data recorded on the recording medium is reproduced by a reproducing device (not shown),
The data is supplied to the buffer 101 and stored.

An IVLC unit (inverse VLC unit) (variable-length decoder) 102 reads out the encoded data stored in the buffer 101 and performs variable-length decoding so that the encoded data is converted into a motion vector Mode, quantization step,
And quantization coefficients. Among them, the motion vector and the prediction mode are supplied to the motion compensator 107,
The quantization step and the quantization coefficient are supplied to the inverse quantizer 103.

The inverse quantizer 103 inversely quantizes the quantized coefficient supplied from the IVLC unit 102 in accordance with the quantization step also supplied from the IVLC unit 102.
The resulting DCT coefficient is output to IDCT unit 104. The IDCT unit 104 performs an inverse DCT on the DCT coefficient from the inverse quantizer 103 and supplies the result to an arithmetic unit 105.

The output of the motion compensator 107 is supplied to the arithmetic unit 105 in addition to the output of the IDCT unit 104. That is, the motion compensator 107 reads the already decoded image stored in the frame memory 106 in accordance with the motion vector and the prediction mode from the IVLC unit 102 as in the case of the motion compensator 41 in FIG. ,
The prediction image is supplied to the arithmetic unit 105. Arithmetic unit 10
5 decodes the original image by adding the signal (prediction residual) from the IDCT unit 104 and the predicted image from the motion compensator 107. This decoded image is stored in the frame memory 1
06 and stored. Note that the IDCT device 104
If the output of is intra-coded,
The output passes through the arithmetic unit 105 and is supplied to and stored in the frame memory 106 as it is.

The decoded image stored in the frame memory 106 is used as a reference image for an image to be subsequently decoded, read out as appropriate, and supplied to, for example, a display (not shown) and displayed.

In MPEG1 and MPEG-2, B pictures are not used as reference pictures, so that the frame memory 41 is used in each of the encoder and the decoder.
(FIG. 42) or 106 (FIG. 43).

In MPEG, in addition to the above-mentioned MP @ ML, various profiles and levels are defined.
In addition, various tools are prepared. One of the typical MPEG tools is scalability, for example.

That is, MPEG introduces a scalable encoding method for realizing scalability corresponding to different image sizes and frame rates. For example, in spatial scalability, when decoding only the bit stream of the lower layer, only an image having a small image size is obtained, and when decoding the bit streams of both the lower layer and the upper layer, an image having a large image size is obtained. .

FIG. 44 shows an example of the configuration of an encoder for realizing spatial scalability. In the spatial scalability, for example, the lower layer corresponds to an image signal having a small image size, and the upper layer corresponds to an image signal having a large image size.

For example, the upper layer coding section 201
The image to be coded is input as it is as the image of the upper layer, and the lower layer coding unit 202 thins out the image to be coded and reduces the number of pixels (accordingly, the resolution is reduced and the size is reduced). Smaller)
Is input as the image of the lower layer.

The lower layer coding section 202 predictively codes the image of the lower layer, for example, as in the case of FIG. 42, and outputs a lower layer bit stream as a result of the coding. Further, the lower layer encoding unit 202 generates a locally decoded image of the lower layer enlarged to the same size as the image of the upper layer (hereinafter, appropriately referred to as an enlarged image). This enlarged image is supplied to the upper layer encoding unit 201.

In the upper layer coding section 201,
For example, as in the case of FIG. 42, the image of the upper layer is predictively encoded, and an upper layer bit stream is output as a result of the encoding. Note that the upper layer encoding unit 201 performs predictive encoding using the enlarged image from the lower layer encoding unit 202 as a reference image.

The upper layer bit stream and the lower layer bit stream are multiplexed and output as encoded data.

FIG. 45 is a diagram showing the lower layer coding unit 2 in FIG.
02 shows an example configuration. Note that, in the figure, parts corresponding to those in FIG. 42 are denoted by the same reference numerals. That is, the lower layer encoding unit 202 is configured similarly to the encoder in FIG. 42 except that an upsampling unit 211 is newly provided.

The up-sampling unit 211 up-samples (interpolates) the locally decoded lower-layer image output from the arithmetic unit 40, thereby enlarging the image to the same image size as the upper-layer image. , Are supplied to the upper layer coding section 201.

FIG. 46 shows the upper layer coding section 2 of FIG.
01 shows an example configuration. Note that, in the figure, parts corresponding to those in FIG. 42 are denoted by the same reference numerals. That is, the upper layer encoding unit 201 is basically configured in the same manner as the encoder of FIG. 42 except that weighting units 221 and 222 and a calculator 223 are newly provided.

The weighting unit 221 multiplies the predicted image output from the motion compensator 42 by a weight W,
Output to 3. The output of the weighting unit 222 is also supplied to the arithmetic unit 223 in addition to the output of the weighting unit 221.
The weighting unit 222 multiplies the enlarged image supplied from the lower layer encoding unit 202 by a weight (1-W),
It is supplied to the arithmetic unit 223.

The arithmetic unit 223 adds the outputs of the weighting circuits 221 and 222, and outputs the addition result to the arithmetic units 33 and 40 as a predicted image.

Thereafter, in upper layer coding section 201, the same processing as in the case of FIG. 42 is performed.

Therefore, in the upper layer coding section 201,
Predictive coding is performed using not only the image of the upper layer as a reference image but also the enlarged image from the lower layer encoding unit 202, that is, the image of the lower layer as a reference image.

The weight W used in the weight adding section 221 is set in advance (therefore, the weight 1-W used in the weight adding section 222 is also set in advance). , VLC unit 36 and are subjected to variable-length coding.

FIG. 47 shows an example of the configuration of a decoder for realizing spatial scalability.

The encoded data output from the encoder of FIG. 44 is separated into an upper layer bit stream and a lower layer bit stream, and each is supplied to the upper layer decoding section 231 or the lower layer decoding section 232. .

In lower layer decoding section 232, the lower layer bit stream is decoded in the same manner as in FIG. 43, and the resulting lower layer decoded image is output. Further, the lower layer decoding unit 232 enlarges the decoded image of the lower layer to the same size as the image of the upper layer, thereby generating an enlarged image. This enlarged image is output to the upper layer decoding unit 231.
Supplied to

In the upper layer decoding section 231,
For example, as in the case of FIG. 43, the upper layer bit stream is decoded. However, the upper layer decoding unit 231 performs decoding using the enlarged image from the lower layer decoding unit 232 as a reference image.

FIG. 48 shows the lower layer decoding unit 2 in FIG.
32 shows an example configuration. Note that, in the figure, parts corresponding to the case in FIG. 43 are denoted by the same reference numerals. That is, the lower layer decoding unit 232 has the same configuration as the decoder in FIG. 43 except that an upsampling unit 241 is newly provided.

The up-sampling unit 241 up-samples (interpolates) the decoded lower-layer image output from the arithmetic unit 105 to enlarge the image to the same image size as the upper-layer image. This is supplied to the upper layer decoding unit 231.

FIG. 49 shows the upper layer decoding section 2 of FIG.
31 shows an example configuration. Note that, in the figure, parts corresponding to the case in FIG. 43 are denoted by the same reference numerals. That is, the upper layer decoding unit 231 is basically configured in the same manner as the encoder in FIG. 43 except that the weighting units 251 and 252 and the arithmetic unit 253 are newly provided.

The IVLC unit 102 extracts the weight W from the encoded data in addition to the processing described with reference to FIG. 43 and outputs the weight W to the weight adding units 251 and 252. The weight adding unit 251 includes:
For the predicted image output from the motion compensator 107, the weight W
And outputs the result to the calculator 253. The output of the weighting unit 252 is also supplied to the arithmetic unit 253 in addition to the output of the weighting unit 251. The weighting unit 252 performs weighting on the enlarged image supplied from the lower layer decoding unit 232. The product is multiplied by (1−W) and supplied to the calculator 253.

The arithmetic unit 253 adds the outputs of the weighting circuits 251 and 252, and outputs the addition result to the arithmetic unit 105 as a predicted image.

As described above, upper layer decoding section 231
Now, in addition to using the upper layer image as a reference image,
Decoding is performed using the enlarged image from the lower layer encoding unit 232, that is, the image of the lower layer as a reference image.

The above-described processing is performed on both the luminance signal and the color difference signal. However, as the motion vector of the color difference signal, for example, a value obtained by halving the motion vector of the luminance signal is used.

At present, in addition to the above-mentioned MPEG system, various high-efficiency coding systems for moving images are standardized. For example, in ITU-T, H.264 is mainly used as an encoding method for communication. 261 and H.E. 263 is defined. This H. 261 and H.E. 263 is basically M
Similar to the PEG method, this is a combination of motion compensation prediction coding and DCT transform coding. Although details such as header information are different, the basic configuration of an encoder and a decoder is as follows.
This is similar to the case of the MPEG system.

[0057]

In an image synthesizing system for synthesizing a plurality of images to form one image, for example, a technique called chroma key is used. In this method, an object is photographed in front of a specific color background such as blue, and the non-blue area is extracted therefrom and combined with another image.The signal indicating the extracted area is a key. Signal (ke
y signal).

FIG. 50 is a diagram for explaining a conventional image synthesizing method. Here, it is assumed that the image F1 is a background and the image F2 is a foreground. The image F2 is obtained by photographing an object (here, a person) in front of a background of a specific color and extracting a region other than the color. The key signal K1 is obtained by extracting the key signal K1. This is a signal indicating an area.

In the image synthesizing system, the background image F
1 and the image F2, which is the foreground, are synthesized according to the key signal K1 to generate a synthesized image F3. The composite image F3 is transmitted after being subjected to, for example, MPEG encoding.

When the composite image F3 is encoded and transmitted as described above, what is transmitted is the composite image F3.
, The key signal K1
Information is lost, so that it becomes difficult for the receiving side to re-edit and re-synthesize the image, for example, changing only the background F1 while keeping the foreground F2.

Therefore, for example, as shown in FIG. 51, a method of independently encoding the images F1 and F2 and the key signal K1 and multiplexing the respective bit streams is conceivable. In this case, on the receiving side, for example, FIG.
As shown in (1), the multiplexed data is demultiplexed to obtain a bit stream of the images F1 and F2 or the key signal K1, and each bit stream is decoded. Then, by performing synthesis using the images F1 and F2 obtained thereby or the decoding result of the key signal K1, a synthesized image F3 is generated. In this case, the receiving side can perform re-editing and re-synthesis, for example, changing the background F1 to another image while leaving the foreground F2 as it is.

Incidentally, the composite image F3 is composed of the images F1 and F
Similarly, any image can be considered to be composed of a plurality of images (objects). Now, assuming that a unit forming an image in this way is called a VO (Video Object), a method of performing encoding in such a VO unit is currently under ISO.
-In IEC / JTC1 / SC29 / WG11, M
Standardization work is underway for PEG4.

However, at present, a method for efficiently encoding VO and a method for encoding key signals have not been established, and are unsolved problems.

Although MPEG4 specifies provision of a scalability function, no specific method for realizing scalability for a VO whose position and size change with time has not been proposed.

That is, for example, when a person or the like coming from a distance is a VO, the position and size of the VO change over time. Therefore, when a lower layer image is used as a reference image in predictive encoding of an upper layer image, the relative positional relationship between the upper layer image and the lower layer image used as the reference image is clearly defined. There is a need to.

When scalability is performed on a VO basis, the condition of a skip macroblock in a lower layer does not always directly apply to the condition of a skip macroblock in an upper layer.

The present invention has been made in view of such a situation, and is intended to easily realize coding on a VO basis.

[0068]

[Means for Solving the Problems]

[0069]

According to a first aspect of the present invention, there is provided an image coding apparatus for coding a macroblock of a B picture by one of forward prediction coding, backward prediction coding, and bidirectional prediction coding. Macro block
I or P picture coded just before being coded
Macro of B picture of macroblocks constituting
The one corresponding to the block is the skip macro block.
B picture is a skip macro block
Is determined .

[0070] Image coding method according to claim 2, forward predictive coding, backward predictive coding, or by any of the bidirectional predictive coding, when encoding a macroblock of a B picture, a macro block Just before is encoded
Macro block that constitutes an I or P picture encoded in
Of the lock, corresponding to the macroblock of the B picture
When the object is a skip macro block,
The character is determined to be a skip macro block .

[0071] The image decoding apparatus according to claim 3, forward predictive coding, backward predictive coding, or by any of the bidirectional predictive coding, when decoding a macroblock of the B picture, macroblock Just before is decrypted
Macroblocks constituting an I or P picture decoded in
Of the lock, corresponding to the macroblock of the B picture
When the object is a skip macro block,
The character is determined to be a skip macro block .

[0072] Image decoding method according to claim 4, forward predictive coding, backward predictive coding, or by any of the bidirectional predictive coding, when decoding a macroblock of the B picture, macroblock Just before is decrypted
Macroblocks constituting an I or P picture decoded in
Of the lock, corresponding to the macroblock of the B picture
When the object is a skip macro block,
The character is determined to be a skip macro block .

[0073]

[0074] recording method according to claim 5, forward predictive coding, backward predictive coding, or by any of the bidirectional predictive coding, when encoding a macroblock of a B picture, macroblock code Sign just before
Macroblocks Constituting a Converted I or P Picture
Of which corresponds to a macroblock of a B picture
Is a skip macroblock, the B picture
Is determined to be a skip macroblock .

In the image encoding apparatus according to the first aspect, the image encoding method according to the second aspect , and the image decoding apparatus according to the third aspect and the image decoding method according to the fourth aspect , When a macroblock of a B picture is coded by one of forward prediction coding, backward prediction coding, and bidirectional prediction coding, the macroblock
I or P picture encoded just before is encoded
Of the B picture of the macroblocks
The one corresponding to the local block is the skip macro block.
At one time, a B picture is a skip macroblock
It has been decided to be.

[0076] In the recording method according to claim 5, forward predictive coding, backward predictive coding, or by any of the bidirectional predictive coding, when encoding a macroblock of a B picture, macroblock Immediately encoded
Macros that make up a previously encoded I or P picture
Supports B-picture macroblocks in blocks
When the skipping macroblock is
Kucha is considered to be a skip macroblock
You.

[0077]

FIG. 1 shows an embodiment of an encoder to which the present invention is applied.

The image data to be coded is
The VO composing unit 1 extracts an object constituting an image inputted thereto and constructs a VO. Furthermore, V
The O component 1 generates a key signal for each VO and outputs the key signal to the VOP components 21 to 2N together with the corresponding VO. That is, in the VO configuration unit 1, N VO1s
When VO # N is configured, the N VO1 to VO # N are output to the VOP constituent units 21 to 2N together with the corresponding key signals.

More specifically, for example, the image data to be encoded includes a background F1, a foreground F2, and a key signal K1, as shown in FIG.
If a composite image can be generated by chroma keying, the VO constructing unit 1
O1 and outputs the key signal K1 to the VOP constructing unit 21 as a key signal of the VO1. Further, the VO component 1 outputs the background F1 to the VOP component 22 as VO2. Note that a key signal is not output for the background because it is unnecessary (it is not generated).

When the image data to be encoded does not include a key signal, for example, an already synthesized image, the VO constructing unit 1 divides the image into regions according to a predetermined algorithm. One or more regions are extracted, and a key signal corresponding to each region is generated. Then, the VO configuration unit 1 sets the sequence of the extracted region as VO, and generates the corresponding key signal together with the corresponding V signal.
Output to the OP constituent unit 2n (where n = 1, 2,..., N).

The VOP constructing unit 2 n constructs a VOP (VO Plane) from the output of the VO constructing unit 1. That is, an object is extracted from each frame, and, for example, a minimum rectangle surrounding the object is set as a VOP. At this time, the VOP forming unit 2n forms the VOP such that the number of horizontal and vertical pixels is a multiple of, for example, 16. The VO component 2n
When a VOP is configured, image data of an object portion included in the VOP (for example, a luminance signal and a color difference signal)
Is output to the VOP encoding unit 3n together with a key signal for extracting the symbol (this key signal is supplied from the VO configuration unit 1 as described above).

Further, the VOP constructing unit 2n includes size data (VOP size) indicating the size of the VOP (for example, horizontal and vertical lengths) and the position of the VOP in the frame (for example, The offset data (VOP offset) representing the origin (coordinates at the origin) is detected, and these data are also supplied to the VOP encoding unit 3n.

The VOP encoder 3n outputs the output of the VOP constructor 2n to, for example, MPEG or H.264. H.263 and the like, and the resulting bit stream is output to the multiplexing unit 4. The multiplexing unit 4 includes a VO
The bit streams from the P encoders 31 to 3N are multiplexed, and the resulting multiplexed data is transmitted via, for example, a terrestrial wave, a satellite line, a CATV network or another transmission path 5, or The information is recorded on a recording medium 6 such as a magnetic disk, a magneto-optical disk, an optical disk, a magnetic tape, or the like.

Here, VO and VOP will be described.

VO is a sequence of each object constituting the composite image when a sequence of the composite image exists, and VOP means VO at a certain time. That is, for example, if there is a composite image F3 composed of the images F1 and F2, the image F1 or F2 arranged in time series is VO, and the image F1 or F2 at a certain time is , Each VOP
It is. Therefore, VO is the V of the same object at different times.
It can be said that it is a set of OPs.

For example, when the image F1 is set as the background and the image F2 is set as the foreground, the composite image F3 is formed by using the key signals for extracting the image F2 and the images F1 and F2.
2, the VOP of the image F2 in this case includes not only the image data (luminance signal and color difference signal) constituting the image F2 but also its key signal as appropriate.

Although the sequence of image frames (image frames) does not change in both size and position, VO
The size and position may change. That is, the same VO
May be different in size and position depending on the time.

More specifically, FIG. 2 shows an image F1 as a background.
And a composite image including the foreground image F2.

The image F1 is, for example, a photograph of a certain natural scenery, and the sequence of the entire image is one VO (referred to as VO0). Also, the image F2
Is an image of a person walking, for example, and the smallest rectangular sequence surrounding the person is one V
O (referred to as VO1).

In this case, since VO0 is a landscape image, basically, both its position and its size do not change, similarly to a normal image frame. On the contrary,
Since the VO1 is an image of a person, the size and position of the VO1 change when the person moves left and right or moves forward or backward in the drawing. Therefore, FIG.
Represents VO0 and VO1 at the same time, but their positions and sizes are not necessarily the same.

Therefore, the VOP encoding unit 3n in FIG. 1 includes, in the output bit stream, information on the position (coordinates) and size of the VOP in a predetermined absolute coordinate system, in addition to the data obtained by encoding the VOP. It has been made like that. In FIG. 2, VO0 at a certain time
The vector indicating the position of (VOP) is OST0, and the vector indicating the position of VO1 (VOP) at the same time is OST0.
ST1.

Next, FIG. 3 shows the VOP encoding section 3n of FIG.
2 shows an example of a basic configuration.

Image signal (image data) from VOP forming section 2n (luminance signal and color difference signal forming VOP)
Is input to the image signal encoding unit 11. The image signal encoding unit 11 is basically configured in the same manner as the encoder of FIG. 42 described above.
EG and H. H.263 and the like. The motion and texture information obtained by encoding the VOP in the image signal encoding unit 11 is supplied to the multiplexing unit 13.

The key signal from the VOP forming unit 2n is input to the key signal encoding unit 12, where, for example, DPCM (Differential Pulse Code Modulation).
And so on. Key signal information obtained as a result of encoding in the key signal encoding unit 12 is as follows:
It is also supplied to the multiplexing unit 13.

The multiplexing unit 13 includes the image signal encoding unit 11
In addition to the output of the key signal encoding unit 12 and the size data (VOP size) and offset data (VOP offset) from the VOP configuration unit 2n, the multiplexing unit 13
Multiplexes these and outputs them to the buffer 14. The buffer 14 temporarily stores the output of the multiplexing unit 13, smoothes the data amount, and outputs the data.

Note that, in the key signal encoding unit 12,
In addition to the DPCM, for example, the key signal is motion-compensated in accordance with the motion vector detected by performing the predictive encoding in the image signal encoding unit 11, and the key signal is compared with the key signal in the temporally preceding or succeeding VOP. By calculating the difference, the key signal can be encoded.

The data amount (buffer feedback) of the encoding result of the key signal in the key signal encoding unit 12 can be supplied to the image signal encoding unit 11. In this case, the quantization step is determined in the image signal encoding unit 11 in consideration of the data amount from the key signal encoding unit 12 as well.

Next, FIG. 4 shows a configuration example of the VOP encoding unit 3n of FIG. 1 for realizing scalability.

The VOP (image data) from the VOP constructing unit 2n, its key signal, and size data (VOP siz
e) and the offset data (VOP offset) are both supplied to the image layering unit 21.

The image layering unit 21 generates a plurality of layers of image data from the VOP (performs VOP layering). That is, for example, when performing spatial scalability encoding, the image layering unit 21 outputs the image data and the key signal input thereto as it is as the image data and the key signal of the upper layer (upper layer). At the same time, the image data and the key signal are reduced by reducing the number of pixels constituting the key signal (reducing the resolution), and are output as lower layer (lower layer) image data and a key signal.

It is also possible to use the input VOP as data of the lower layer, increase the resolution of the VOP by some method (increase the number of pixels), and use this as data of the upper layer. It is.

Although the number of layers can be three or more, the case of two layers will be described here for simplicity.

For example, when performing temporal scalability (temporal scalability) encoding, the image layering unit 21 alternately converts image data and key signals as lower layer or upper layer data, for example, according to time. Output. That is, for example, the image layering unit 21
Indicates that VOPs constituting a certain VO are VOP0,
If it is assumed that VOP1, VOP2, VOP3,... Are input in this order, VOP0, VOP2, VOP4, VOP
, ... as lower layer data, and VO
P1, VOP3, VOP5, VOP7,... Are output as upper layer data. In the case of time scalability, VOPs are thinned out in this way,
Only the lower layer and upper layer data,
The enlargement or reduction (resolution conversion) of the image data is not performed (however, it can be performed).

Further, the image hierarchical unit 21 can, for example,
When performing R (Signal to Noise Ratio) scalability encoding, input image data and a key signal are output as data of an upper layer or a lower layer as they are. That is, in this case, the image data and the key signal of the lower layer and the upper layer are the same data.

Here, the following three types of spatial scalability when encoding is performed for each VOP can be considered, for example.

That is, for example, as a VOP, FIG.
Assuming that a composite image composed of the images F1 and F2 as shown in FIG. 5 is input, the first spatial scalability is as shown in FIG. 5 as a whole of the input VOP (FIG. 5A).
Is the upper layer (EnhancementLayer), and the reduced VOP (FIG. 5B) is the lower layer (Base Layer).

As shown in FIG. 6, the second spatial scalability is such that some objects (FIG. 6A (here, a portion corresponding to image F2)) constituting the input VOP are used. Extract (Note that such extraction is
For example, the processing is performed in the same manner as in the VOP composing unit 2n, so that the extracted object can also be considered as one VOP), the upper layer, and a reduced VOP as a whole ( FIG. 6B) is a lower layer.

Furthermore, the third spatial scalability is:
As shown in FIGS. 7 and 8, an object (VOP) constituting an input VOP is extracted, and an upper layer and a lower layer are generated for each object. FIG. 7 shows a background (image F1) constituting the VOP of FIG.
8 shows a case where an upper layer and a lower layer are generated from the foreground (image F2) constituting the VOP of FIG. 2.

Which of the above scalabilities is used is determined in advance, and the image hierarchy unit 2
1 hierarchically arranges VOPs so that encoding based on the predetermined scalability can be performed.

Further, the image layering unit 21 generates the lower layer and the upper layer generated from the VOP size data and the offset data (hereinafter referred to as initial size data and initial offset data, respectively, as appropriate). The offset data indicating the position of the VOP in the predetermined absolute coordinate system and the size data indicating the size are calculated (determined).

Here, V of the lower layer and the upper layer
A method of determining the offset data (position information) and the size data of the OP will be described by taking, for example, a case where the above-described second scalability (FIG. 6) is performed.

In this case, the offset data FPOS_B of the lower layer is obtained by enlarging (interpolating) the image data of the lower layer based on the difference between the resolution of the lower layer and the resolution of the upper layer, as shown in FIG. 9A, for example. Sometimes, that is, the enlargement ratio of the image of the lower layer to match the size of the image of the upper layer (the reciprocal of the reduction ratio when the image of the lower layer is generated by reducing the image of the upper layer)
(Hereinafter, appropriately referred to as magnification FR), the offset data of the enlarged image in the absolute coordinate system is
It is determined to match the initial offset data. Similarly, the size data FSZ_B of the lower layer is determined so that the size data of the enlarged image obtained when the image of the lower layer is enlarged by the magnification FR matches the initial size data.

On the other hand, upper layer offset data FP
OS_E is, for example, as shown in FIG. 9B, the coordinates of, for example, the upper left vertex of the 16-fold minimum rectangle (VOP) surrounding the object extracted from the input VOP is obtained based on the initial offset data. And is determined to this value. The size data FPOS_E of the upper layer is
The length is determined to be, for example, the horizontal and vertical lengths of a 16-fold minimum rectangle surrounding the object extracted from the input VOP.

Accordingly, in this case, the offset data FPOS_B and the size data FPOS_B of the lower layer
Is converted according to the magnification FR (the offset data FPOS_B or the size data FPOS_B after the conversion).
Are respectively referred to as conversion offset data FPOS_B or conversion size data FPOS_B), and an image frame having a size corresponding to the conversion size data FSZ_B is considered at a position corresponding to the conversion offset data FPOS_B in the absolute coordinate system. F is the image data of the layer
In addition to arranging an enlarged image R times (see FIG. 9)
(A)), the offset data FPOS_E and the size data FPOS of the upper layer in the absolute coordinate system.
If the image of the upper layer is similarly arranged according to _E (FIG. 9B), each pixel constituting the enlarged image and each pixel constituting the image of the upper layer have the same position at the same position. Will be placed in That is, in this case, for example, in FIG. 9, the part of the person, which is the image of the upper layer, and the part of the person in the enlarged image are arranged at the same position.

Similarly, in the first and third scalabilities, the corresponding pixels constituting the enlarged image of the lower layer and the image of the upper layer are arranged at the same position in the absolute coordinate system. Then, offset data FPOS_B and FPOS_E and size data FSZ_B and FSZ_E are determined.

The offset data FPOS_B and FPOS_E and the size data FSZ_B and FSZ_E can be determined as follows, for example.

That is, the offset data FP of the lower layer
OS_B can be determined such that, for example, as shown in FIG. 10A, the offset data of the enlarged image of the lower layer matches a predetermined position in the absolute coordinate system, such as the origin.

On the other hand, upper layer offset data FP
OS_E is, for example, as shown in FIG. 10B, the coordinates of, for example, the upper left vertex of the 16-fold minimum rectangle surrounding the object extracted from the input VOP are obtained based on the initial offset data. The value can be determined by subtracting the initial offset data from the coordinates.

In FIG. 10, the size data FSZ_B of the lower layer and the size data FPOS_E of the upper layer are determined in the same manner as in FIG.

As described above, the offset data FPOS_
Even when B and FPOS_E are determined, an enlarged image of the lower layer and an image of the upper layer are formed.
The corresponding pixels are arranged at the same position in the absolute coordinate system.

Returning to FIG. 4, the image data, the key signal, the offset data FPOS_E, and the size data FSZ_E of the upper layer generated in the image layering unit 21 are obtained.
Is a delay circuit 22, and a lower layer coding unit 25 described later
Is delayed by the processing time in the upper layer encoding unit 2
3 is supplied. The lower layer image data, the key signal, the offset data FPOS_B, and the size data FSZ_B are supplied to the lower layer encoding unit 25. Further, the magnification FR is supplied to the upper layer encoding unit 23 and the resolution conversion unit 24 via the delay circuit 22.

In lower layer coding section 25, the lower layer image data (second image) and the key signal are coded, and the resulting coded data (bit stream) has offset data FPOS_B and size data FSZ_B. And supplied to the multiplexing unit 26.

The lower layer encoding unit 25 locally decodes the encoded data, and outputs the image data of the lower layer as a result of the local decoding to the resolution conversion unit 24. The resolution conversion unit 24 returns the image data of the lower layer from the lower layer encoding unit 25 to the original size by enlarging (or reducing) the image data according to the magnification FR.
The enlarged image obtained in this way is converted into an upper layer encoding unit 2
Output to 3.

On the other hand, the upper layer encoding section 23 encodes the image data (first image) and the key signal of the upper layer, and adds the offset data FPOS_E and the size to the resulting encoded data (bit stream). The data FSZ_E is included and supplied to the multiplexing unit 26. In the upper layer encoding unit 23, encoding of the upper layer image data is performed using the enlarged image supplied from the resolution conversion unit 24 as a reference image.

In the multiplexing section 26, the upper layer coding section 2
3 and the output of the lower layer coding unit 25 are multiplexed and output.

Note that the lower layer encoder 25 is supplied with the lower layer size data FSZ_B, offset data FPOS_B, motion vector MV, flag COD, etc., from the upper layer encoder The conversion unit 23 performs processing while referring to these data as necessary. The details will be described later.

Next, FIG. 11 shows a detailed configuration example of the lower layer coding unit 25 of FIG. Note that in FIG.
The same reference numerals are given to the portions corresponding to the case in. That is, the lower layer encoding unit 25 is basically configured in the same manner as the encoder in FIG. 42 except that a key signal encoding unit 43 and a key signal decoding unit 44 are newly provided.

The image data from the image layering unit 21 (FIG. 4), that is, the VOP of the lower layer is supplied to and stored in the frame memory 31 as in the case of FIG.
In the motion vector detector 32, a motion vector is detected for each macroblock.

However, the motion vector detector 32 of the lower layer encoding unit 25 is supplied with the size data FSZ_B and the offset data FPOS_B of the VOP of the lower layer, where the size data FSZ_B and the offset data FPOS_B are supplied. A motion vector of a macroblock is detected based on the offset data FPOS_B.

That is, as described above, the size and the position of the VOP change depending on the time (frame).
In detecting the motion vector, it is necessary to set a coordinate system serving as a reference for the detection and to detect a motion in the coordinate system. Therefore, here, the motion vector detector 43 uses the above-described absolute coordinate system as a reference coordinate system, and uses the absolute coordinate system as a reference according to the size data FSZ_B and the offset data FPOS_B. Place a VOP to
A motion vector is detected.

Further, the motion vector detector 32 encodes the key signal of the lower layer, and supplies a decoded key signal obtained by decoding the encoded result from the key signal decoding unit 44. The motion vector detector 32
With this decryption key signal, an object is extracted from the VOP and its motion vector is detected.
Here, the reason why the decoding key signal is used instead of the original key signal (key signal before encoding) to extract the object is that the decoding key signal is used on the receiving side.

Note that the detected motion vector (MV)
Is supplied to the VLC unit 36 and the motion compensator 42 together with the prediction mode, and is also supplied to the upper layer encoding unit 23 (FIG. 4).

Also, when performing motion compensation, it is necessary to detect the motion in the reference coordinate system, as described above. Therefore, the size data FSZ_B and the offset data FPOS_
B is supplied. Further, the decoded key signal is supplied from the key signal decoding unit 44 to the motion compensator 42 for the same reason as in the motion vector detector 32.

The VOP in which the motion vector is detected is shown in FIG.
2, and is supplied to the VLC unit 36 as quantized data. As in the case of FIG. 42, the VLC unit 36 also includes quantized data, a quantized step,
In addition to the motion vector and the prediction mode, size data FSZ_B and offset data FPOS_B from the image layering unit 21 are also supplied, where all of these data are variable-length coded. Further, the VLC unit 36 is also supplied with the encoding result of the key signal (the bit stream of the key signal) from the key signal encoding unit 43. The VLC unit 36 outputs the encoding result of the key signal. Is also output after variable-length encoding.

That is, the key signal encoding unit 43 encodes the key signal from the image layering unit 21 as described with reference to FIG. 3, for example, and outputs the encoded result to the VLC unit 36. The encoding result of the key signal is also supplied to the key signal decoding unit 44 in addition to the VLC unit 36, and the key signal decoding unit 44 decodes the encoding result of the key signal and outputs the decoded key signal. (Decryption key signal) is output to the motion vector detector 32, the motion compensator 42, and the resolution converter 24 (FIG. 4).

Here, in addition to the key signal of the lower layer, the size signal FSZ_B and the offset data FPOS_B are supplied to the key signal encoding unit 43. Similarly, the position and range of the key signal in the absolute coordinate system are recognized based on the data.

The VOP in which the motion vector has been detected is coded as described above, and is also locally decoded as in FIG. 42 and stored in the frame memory 41. The decoded image is used as a reference image as described above, and is output to the resolution conversion unit 24.

Note that in MPEG4, MPEG1
Unlike B and 2, the B picture is also used as a reference picture, so the B picture is also locally decoded and stored in the frame memory 41 (however, at this time, the B picture is used as a reference picture). Only for the upper layers).

On the other hand, as described with reference to FIG. 42, the VLC unit 36 determines whether or not macroblocks of I, P, and B pictures are to be skipped macroblocks.
The flags COD and MODB indicating the determination result are set. The flags COD and MODB are also transmitted after being variable-length coded. Further, the flag COD is also supplied to the upper layer encoding unit 23.

Next, FIG. 12 shows an example of the configuration of the upper layer coding section 23 of FIG. Note that, in the figure, parts corresponding to those in FIG. 11 or FIG. 42 are denoted by the same reference numerals. That is, the upper layer encoding unit 23
Except that a key signal encoding unit 51, a frame memory 52, and a key signal decoding unit 53 are newly provided,
Basically, the configuration is the same as that of the lower layer encoding unit 25 in FIG. 11 or the encoder in FIG.

The image data from the image layering unit 21 (FIG. 4), that is, the VOP of the upper layer is supplied to and stored in the frame memory 31 as in the case of FIG.
In the motion vector detector 32, a motion vector is detected for each macroblock. In this case,
As in the case of FIG. 11, the motion vector detector 32 includes, in addition to the VOP of the upper layer, the size data FS thereof.
When the Z_E and the offset data FPOS_E are supplied, the decryption key is supplied from the key signal decryption unit 53. In the motion vector detector 32,
Similarly to the above case, based on the size data FSZ_E and the offset data FPOS_E, the arrangement position of the VOP of the upper layer in the absolute coordinate system is recognized, and the extraction of the object included in the VOP is included in the decryption key signal. Based on this, a motion vector of a macroblock is detected.

Here, the motion vector detector 32 in the upper layer encoding section 23 and the lower layer encoding section 25
Then, as described with reference to FIG. 42, the VOP is processed according to a predetermined sequence that is set in advance. Here, the sequence is set as follows, for example.

That is, in the case of spatial scalability, as shown in FIG. 13A or FIG.
The VOP of the upper layer or the lower layer is, for example, P,
Are processed in the order of B, B, B,... Or I, P, P, P,.

Then, in this case, the first V
The P picture that is the OP is encoded using, for example, a VOP (here, an I picture) of a lower layer at the same time as a reference image. The B picture that is the second or later VOP of the upper layer is, for example, the VOP of the immediately preceding upper layer and the VOP of the lower layer at the same time as the VOP.
The encoding is performed using the OP as a reference image. That is, here, the B picture in the upper layer is used as a reference image when encoding another VOP, like the P picture in the lower layer.

For the lower layer, for example, M
PEG 1 or 2, or H.264. Encoding is performed as in the case of H.263.

Since the SNR scalability is considered to be when the magnification FR in the spatial scalability is 1, it is processed in the same manner as in the case of the spatial scalability described above.

In the case of temporal scalability, that is, for example, as described above, VO is VOP0, VO
P1, VOP2, VOP3,...
, VOP3, VOP5, VOP7,... Are upper layers (FIG. 14A), and VOP0, VOP2, VO
When P4, VOP6,... Are the lower layers (FIG. 14B), as shown in FIG. 14, the VOPs of the upper layer or the lower layer are, for example, B, B,
B,... Or I, P, P, P,.

Then, in this case, the first V
OP1 (B picture) is, for example, a lower layer VOP.
0 (I picture) and VOP2 (P picture) are coded using as reference pictures. Also, the upper layer 2
The VOP3 (B-picture) is, for example, VOP1 of the upper layer coded immediately before as a B-picture,
And VOP4 (P picture) of the lower layer, which is an image at the time (frame) next to VOP3, is used as a reference image. Third VOP of upper layer
5 (B picture), like VOP3, for example, the VO of the upper layer coded immediately before as a B picture
Encoding is performed using P3 and VOP6 (P picture) of the lower layer, which is an image at the time (frame) next to VOP5, as a reference image.

As described above, with respect to a VOP of a certain layer (here, an upper layer), another layer (a scalable layer) (here, a lower layer) is used as a reference image for coding P and B pictures. VOP can be used. As described above, when a VOP of another layer is used as a reference image to encode a VOP of a certain layer, that is, here, a VOP of a lower layer is referred to for predictive encoding of a VOP of an upper layer. When used as an image, the motion vector detector 32 of the upper layer encoding unit 23 (FIG. 12) outputs a flag ref_
A layer_id (when there are three or more layers, the flag ref_layer_id indicates a layer to which a VOP used as a reference image belongs) is set and output.

Further, the motion vector detector 32 of the upper layer encoding unit 23 sets the flag ref_
According to the layer_id, a flag ref_select__ indicating which layer VOP is to be used as a reference image for forward prediction coding or backward prediction coding, respectively.
A code (reference image information) is set and output.

That is, FIG. 15 (A) or (B) shows a flag ref_select for a P or B picture.
_Code are shown respectively.

For example, the upper layer (Enhancement Layer)
When the P picture of r) is encoded using a VOP belonging to the same layer as the reference picture, which is decoded immediately before (local decoding), the flag ref_selec
t_code is set to “00”. Also, the VO belonging to a different layer (here, lower layer) (Reference Layer) displayed immediately before the P picture is displayed.
If encoding is performed using P as a reference image, the flag r
ef_select_code is set to “01”. Further, when the P picture is encoded using a VOP that is displayed immediately after and belongs to a different layer from the POP as a reference image, the flag ref_select_code is used.
Is set to “10”. When a P picture is encoded using a VOP of a different layer at the same time as the reference picture as a reference picture, the flag ref_select_
The code is set to “11” (FIG. 15A).

On the other hand, for example, a BOP in an upper layer uses a VOP of a different layer at the same time as a reference picture for forward prediction, and decodes immediately before the VOP belonging to the same layer. Is encoded using as a reference image for backward prediction,
The flag ref_select_code is set to “00”. In addition, a B-picture of an upper layer uses a VOP belonging to the same layer as a reference image for forward prediction, and displays a VOP belonging to a different layer, which is displayed immediately before and belongs to a different layer, as a reference image for backward prediction. When the encoding is performed by using the flag ref_select
_Code is set to “01”. Further, the BOP of the upper layer is decoded immediately before and uses the VOP belonging to the same layer as the reference image for forward prediction, and the VOP belonging to a different layer displayed immediately after that and belonging to a different layer is backward. If the encoding is performed using a reference image for prediction, the flag ref_select_c is used.
mode is set to “10”. Further, a VOP belonging to a different layer, which is displayed immediately before the B picture of the upper layer and belongs to a different layer, is used as a reference image for forward prediction,
In the case where encoding is performed using a VOP displayed immediately after that and belonging to a different layer as a reference image for backward prediction, a flag ref_select_code is used.
Is set to “11” (FIG. 15B).

Here, the method of predictive coding described with reference to FIGS. 13 and 14 is one example.
Which VOP of which layer is used as a reference image in backward prediction coding or bidirectional prediction coding is as follows.
For example, it can be set freely within the range described with reference to FIG.

In the above case, for convenience,
Although the terms “spatial scalability”, “temporal scalability”, and “SNR scalability” are used, as described with reference to FIG. 15, when a reference image used for predictive coding is set, that is, as shown in FIG. When the syntax is used, the flag ref_select_c
The mode makes it difficult to clearly distinguish spatial scalability, temporal scalability, and SNR scalability. That is, conversely, the flag r
By using ef_select_code,
It is not necessary to distinguish the scalability as described above.

The scalability and the flag r
If ef_select_code is associated, for example, it is as follows. That is, for the P picture, the flag ref_select_code is set to “1”.
Since the case of “1” is a case where the VOP of the layer indicated by the flag ref_layer_id at the same time is used as a reference image (a reference image for forward prediction), this corresponds to spatial scalability or SNR scalability. Then, the flag ref_select_
Except when the code is “11”, it corresponds to temporal scalability.

For a B picture, the flag re
Since the case where f_select_code is “00” is the case where the VOP of the layer indicated by the flag ref_layer_id at the same time is used as a reference image for forward prediction, this corresponds to spatial scalability or SNR scalability. Except when the flag ref_select_code is “00”, it corresponds to temporal scalability.

It is to be noted that a different layer (here, a lower layer) is used for predictive coding of the VOP of the upper layer.
When the VOP at the same time is used as a reference image, there is no motion between the two and the motion vector is always 0 (0, 0).

Returning to FIG. 12, in the motion detector 31 of the upper layer coding unit 23, the flag ref_la
Yer_id and ref_select_code are set and supplied to the motion compensator 42 and the VLC unit 36.

In the motion vector detector 32, the flags ref_layer_id and ref_select
According to t_code, a motion vector is detected not only by referring to the frame memory 31 but also by referring to the frame memory 52 as necessary.

Here, the frame memory 52 is supplied with the locally decoded enlarged image of the lower layer from the resolution converter 24 (FIG. 4). That is, in the resolution conversion unit 24, the locally decoded VOP of the lower layer
Is enlarged by, for example, a so-called interpolation filter, thereby generating an enlarged image obtained by multiplying the VOP by FR times, that is, an enlarged image having the same size as the VOP of the upper layer corresponding to the VOP of the lower layer. Then, it is supplied to the upper layer coding unit 23. The frame memory 52 stores the enlarged image supplied from the resolution conversion unit 24 in this way.

Therefore, when the magnification FR is 1, the resolution conversion unit 24 does not perform any processing on the locally decoded VOP from the lower layer encoding unit 25 without any processing. 23.

The motion vector detector 32 is supplied with the size data FSZ_B and the offset data FPOS_B from the lower layer coding unit 25 and the magnification FR from the delay circuit 22 (FIG. 4). When the motion vector detector 31 uses the enlarged image stored in the frame memory 52 as a reference image, that is, the motion vector detector 31 uses the V
When a lower layer VOP at the same time as the OP is used as a reference image (in this case, as described in FIG.
The flag ref_select_code is set to “11” for the P picture (FIG. (A)) and “00” for the B picture (FIG. (B)), and the size data FSZ_B and offset data corresponding to the enlarged image FPOS_B is multiplied by the magnification FR. Then, based on the multiplication result, the position of the enlarged image in the absolute coordinate system is recognized, and a motion vector is detected.

The motion vector detector 32 is supplied with the motion vector of the lower layer and the prediction mode, and is used in the following case.
That is, for example, the motion vector detection unit 32 sets the flag ref_select_
When the code is “00”, the magnification FR is 1
, Ie, when the SNR is scalable (however, in this case, since the VOP of the upper layer is used for predictive coding of the upper layer, the SN
The R scalability is different from that specified in MPEG2). Since the upper layer and the lower layer are the same image, the predictive coding of the B picture of the upper layer
The motion vector and the prediction mode of the image at the same time of the lower layer can be used as they are. Therefore, in this case, the motion vector detection unit 32 does not particularly perform processing on the B picture of the upper layer, and adopts the motion vector and the prediction mode of the lower layer as they are.

In this case, upper layer coding section 23
Then, from the motion vector detector 32 to the VLC unit 36,
The motion vector and prediction mode are not output (and therefore not transmitted). This is because the receiving side can recognize the motion vector and the prediction mode of the upper layer from the decoding result of the lower layer.

As described above, the motion vector detector 32
Detects a motion vector using an enlarged image as a reference image in addition to the VOP of the upper layer, and sets a prediction mode that minimizes a prediction error (or variance) as described with reference to FIG. . Also, the motion vector detector 3
2 is, for example, a flag ref_select_code
And ref_layer_id and other necessary information are set and output.

In FIG. 12, lower layer coding section 2
5, a flag COD indicating whether a macroblock constituting an I or P picture in a lower layer is a skip macroblock is determined by the motion vector detector 3
2, which is supplied to the VLC unit 36 and the motion compensator 42, which will be described later.

The macroblock in which the motion vector has been detected is coded in the same manner as described above.
The LC unit 36 outputs a variable-length code as a result of the encoding.

The VLC unit 36 of the upper layer coding unit 23 sets and outputs flags COD and MODB as in the case of the lower layer coding unit 25. Here, the flag COD indicates whether the macroblock of the I or P picture is a skip macroblock, as described above. The flag MODB indicates whether the macroblock of the B picture is a skip macroblock. It indicates whether or not.

The VLC unit 36 has a quantization coefficient, a quantization step, a motion vector, and a prediction mode.
Magnification FR, flag ref_select_code, r
The ef_layer_id, the size data FSZ_E, the offset data FPOS_E, and the output of the key signal encoding unit 51 are also supplied. The VLC unit 36 performs variable-length encoding on all of these data and outputs them.

On the other hand, the macroblock in which the motion vector has been detected is encoded, then locally decoded as described above, and stored in the frame memory 41. Then, in the motion compensator 42, similarly to the case of the motion vector detector 32, not only the locally decoded VOP of the upper layer stored in the frame memory 41 but also the local decoding The motion compensation is performed using the VOP of the lower layer enlarged as a reference image, and a predicted image is generated.

That is, in addition to the motion vector and the prediction mode, the flag ref_select_
code, ref_layer_id, decryption key signal,
The magnification FR, the size data FSZ_B, FSZ_E, and the offset data FPOS_B, FPOS_E are supplied, and the motion compensator 42 sets the flag ref
_Select_code, ref_layer_id
, The reference image to be motion-compensated is recognized, and the VO of the locally decoded upper layer is used as the reference image.
When using P or the enlarged image, the position and size in the absolute coordinate system are recognized based on the size data FSZ_E and the offset data FPOS_E, or the size data FSZ_B and the offset data FPOS_B. A predicted image is generated using the FR and the decryption key signal.

On the other hand, the key signal of the VOP of the upper layer is
It is supplied to the key signal encoding unit 51. In the key signal encoding unit 51, for example, the key signal is encoded in the same manner as in the case of the key signal encoding unit 43 in FIG.
Is supplied to the device 36 and the key signal decoding unit 53. In the key signal decoding unit 53, the result of encoding the key signal by the key signal encoding unit 51 is decoded. The decoded key signal is supplied to the motion vector detector 32 and the motion compensator 42 as described above, and is used for extracting the VOP of the upper layer.

Next, FIG. 16 shows a configuration of an embodiment of a decoder for decoding a bit stream output from the encoder of FIG.

Output from the encoder of FIG.
Is received by a receiving device (not shown), or the bit stream recorded on the recording medium 6 is reproduced by a reproducing device (not shown),
The signal is supplied to the demultiplexing unit 71.

The demultiplexer 71 converts the bit stream input thereto into a bit stream VO for each VO.
, VO2,..., And the corresponding V
This is supplied to the OP decoding unit 72n. The VOP decoding unit 72n converts the bit stream from the demultiplexing unit 71 into a VO
(Image data), key signal, size data (VOP size), and offset data (VOP offs)
et) is decoded and supplied to the image reconstruction unit 73.

In the image reconstruction unit 73, the VOP decoding unit 72
An original image is reconstructed based on the output from each of 1 to 72N. The reconstructed image is supplied to the monitor 74 and displayed, for example.

FIG. 17 is a block diagram of the VOP decoding unit 7 shown in FIG.
2n shows a basic configuration example.

The bit stream from demultiplexing section 71 (FIG. 16) is input to demultiplexing section 81, where key signal information and motion and texture information are extracted. Then, the key signal information is supplied to the key signal decoding unit 82, and the motion and texture information is supplied to the image signal decoding unit 83. Further, in the demultiplexing unit 71,
Size data (VOP size) and offset data (VOP offset) are extracted from the bit stream input thereto and supplied to the image reconstruction unit 73 (FIG. 16).

The key signal decoder 82 or the image signal decoder 83 decodes the key signal information or the motion and texture information, respectively, and obtains the resulting key signal or VOP image data (luminance signal and color difference signal). Is supplied to the image reconstruction unit 73.

When the key signal is encoded by the key signal encoder 12 shown in FIG. 3 by performing motion compensation on the key signal according to the motion vector detected by the image signal encoder 11, Image signal decoding unit 83
Are supplied to the key signal decoding unit 82, and the key signal decoding unit 82 decodes the key signal using the motion vector.

Next, FIG. 18 shows a configuration example of the VOP decoding unit 72n of FIG. 16 for realizing scalability.

The bit stream supplied from the demultiplexing section 71 (FIG. 16) is input to the demultiplexing section 91, where the bit stream is separated into an upper layer VOP bit stream and a lower layer VOP bit stream. Is done. The bit stream of the VOP of the upper layer is supplied to the upper layer decoding unit 93 after being delayed by the processing time of the lower layer decoding unit 95 in the delay circuit 92, and the bit stream of the VOP of the lower layer is This is supplied to the layer decoding unit 95.

The lower layer decoding section 95 decodes the bit stream of the lower layer, and supplies the resulting decoded image and key signal of the lower layer to the resolution conversion section 94. Further, the lower layer decoding unit 95 outputs size data FSZ_B, offset data FPOS_B, and the like obtained by decoding the bit stream of the lower layer.
Information necessary for decoding a higher layer VOP, such as a motion vector (MV), a prediction mode, and a flag COD,
This is supplied to the upper layer decoding unit 93.

In the upper layer decoding section 93, the delay circuit 92
The upper layer bit stream provided via
The output is decoded by referring to the outputs of the lower layer decoding unit 95 and the resolution conversion unit 94 as necessary, and the resulting decoded image, key signal, size data FSZ_E, and offset data FPOS_E of the upper layer are output. . Further, the upper layer decoding unit 93 outputs the magnification FR obtained by decoding the bit stream of the upper layer to the resolution conversion unit 94. The resolution conversion unit 94 converts the decoded image of the lower layer using the magnification FR from the upper layer decoding unit 93 in the same manner as in the case of the resolution conversion unit 24 in FIG. The enlarged image obtained by this conversion is supplied to the upper layer decoding unit 93, and is used for decoding the bit stream of the upper layer as described above.

Next, FIG. 19 shows a configuration example of the lower layer decoding section 95 in FIG. Note that, in the figure, parts corresponding to those in the decoder in FIG. 43 are denoted by the same reference numerals. That is, the lower layer decoding unit 95 is different from the one shown in FIG.
3 is basically the same as the decoder of FIG.

[0187] The bit stream of the lower layer from the demultiplexing section 91 is supplied to the buffer 101 and stored. The IVLC unit 102 appropriately reads out a bit stream from the buffer 101 in accordance with the processing state of the subsequent block, and performs variable length decoding on the bit stream to obtain a quantization coefficient, a motion vector, a prediction mode, a quantization mode, Steps, encoded data of key signals, size data FSZ_B, offset data FPOS_B, flag COD, and the like are separated. The quantization coefficient and the quantization step are supplied to the inverse quantizer 103, and the motion vector and the prediction mode are supplied to the motion compensator 107 and the upper layer decoding unit 93 (FIG. 18). The size data FSZ_B and the offset data FPOS_B are supplied to the motion compensator 107, the key signal decoding unit 108, the image reconstruction unit 73 (FIG. 16), and the upper layer decoding unit 93, and the flag COD is set to the upper layer decoding unit. 93. Further, the encoded data of the key signal is supplied to the key signal decoding unit 108.

In the inverse quantizer 103, the IDCT unit 104, the arithmetic unit 105, the frame memory 106, or the motion compensator 107, the inverse quantizer 38, the IDCT unit 39, and the arithmetic unit of the lower layer encoding unit 25 in FIG. 40, the frame memory 41, or the motion compensator 42, the same processing is performed, whereby the lower layer VOP is decoded, and the image reconstruction unit 73, the upper layer decoding unit 93, and the resolution conversion unit 94 ( 18).

In the key signal decoding section 108, the key signal decoding section 44 of the lower layer coding section 25 in FIG.
Is performed, the encoded data of the key signal is decoded, and the resulting key signal is supplied to the image reconstruction unit 73, the upper layer decoding unit 93, and the resolution conversion unit 94. You.

Next, FIG. 20 shows a configuration example of the upper layer decoding section 93 of FIG. Note that, in the figure, parts corresponding to the case in FIG. 43 are denoted by the same reference numerals. That is, the upper layer decoding unit 93 is basically configured in the same manner as the encoder in FIG. 43 except that a key signal decoding unit 111 and a frame memory 112 are newly provided.

The bit stream of the higher layer from the demultiplexer 91 is supplied to the IVLC unit 10 via the buffer 101.
2 is supplied. The IVLC unit 102 performs variable-length decoding on the bit stream of the upper layer to obtain a quantization coefficient, a motion vector, a prediction mode, a quantization step, encoded data of a key signal, size data FSZ_E, offset data FPOS_E, and a scaling factor FR. , Flag ref_l
ayer_id, ref_select_code, C
OD, MODB, etc. are separated. As in the case of FIG. 19, the quantization coefficient and the quantization step are supplied to the inverse quantizer 103, and the motion vector and the prediction mode are supplied to the motion compensator 107. The size data FSZ_E and the offset data FPOS_E are
The signals are supplied to the motion compensator 107, the key signal decoding unit 108, and the image reconstructing unit 73 (FIG. 16).
ODB, ref_layer_id, and ref_s
The select_code is supplied to the motion vector detector 107. Further, the encoded data of the key signal is supplied to the key signal decoding unit 111, and the magnification FR is supplied to the motion compensator 107 and the resolution conversion unit 94 (FIG. 18).

In addition to the data described above, the motion compensator 107 receives the motion vector, flag COD, and size data F of the lower layer from the lower layer decoding unit 95 (FIG. 18).
SZ_B and offset data FPOS_B are supplied. Also, the frame memory 11
2 is supplied with an enlarged image from the resolution conversion unit 94.

Inverse quantizer 103, IDCT unit 104, arithmetic unit 105, frame memory 106, motion compensator 10
7 or the frame memory 112, the inverse quantizer 38, the IDCT unit 39,
By performing the same processing as in the arithmetic unit 40, the frame memory 41, the motion compensator 42, or the frame memory 52, the VOP of the upper layer is decoded and supplied to the image reconstruction unit 73.

The key signal decoding section 111 also includes the key signal decoding section 53 of the upper layer coding section 23 shown in FIG.
Is performed, the encoded data of the key signal is decoded, and the resulting key signal is supplied to the image reconstruction unit 73.

Here, a VO having upper layer decoding section 93 and lower layer decoding section 95 configured as described above is provided.
In the P decoding unit 72n, the decoded image, the key signal, the size data FSZ_E, and the offset data FPOS_E (hereinafter, all of which are appropriately referred to as upper layer data) for the upper layer and the upper layer for the lower layer Image, key signal, size data FSZ_B, and offset data FPO
S_B (hereinafter referred to as “lower layer data” as appropriate, including all of them) is obtained. The image reconstructing unit 73 reconstructs an image from the upper layer data or the lower layer data as follows, for example. It has been made to be.

That is, for example, when the first spatial scalability (FIG. 5) is performed (when the entire input VOP is set as the upper layer and a reduced version of the VOP is set as the lower layer) When both the lower layer data and the upper layer data are decoded, the image reconstructing unit 73 generates a decoded image (VOP) of the upper layer having a size corresponding to the size data FSZ_E based on only the upper layer data. ,If necessary,
Extracted by the key signal, the offset data FPOS_
Place at the location indicated by E. Further, for example, when an error occurs in the bit stream of the upper layer, or when the monitor 74 supports only low-resolution images, decoding of only the lower layer data is performed. Extracts a decoded image (VOP) of the upper layer having a size corresponding to the size data FSZ_B based on only the lower layer data, if necessary, with the key signal and arranges the decoded image at the position indicated by the offset data FPOS_B. .

Further, for example, when the second spatial scalability (FIG. 6) is performed (a part of the input VOP is set as an upper layer, and a reduced whole of the VOP is set as a lower layer). In the case, when both the lower layer data and the upper layer data are decoded, the image reconstructing unit 73 outputs the size data F
A decoded image of a lower layer having a size corresponding to SZ_B is
The image is enlarged according to the magnification FR, and an enlarged image is generated. Further, the image reconstructing unit 73 outputs the offset data F
POS_B is multiplied by FR, and the enlarged image is arranged at a position corresponding to the value obtained as a result. Then, the image reconstruction unit 7
No. 3 arranges the decoded image of the upper layer of the size corresponding to the size data FSZ_E at the position indicated by the offset data FPOS_E.

In this case, the decoded image portion of the upper layer is displayed with a higher resolution than the other portions.

When the decoded image of the upper layer is arranged, the decoded image and the enlarged image are combined, and this combination is performed using the key signal of the upper layer.

Although not shown in FIG. 18 (FIG. 16), the upper layer decoding unit 93 (VOP decoding unit 72n) sends the image reconstruction unit 73
The magnification FR is also supplied, and the image reconstruction unit 73 generates an enlarged image using the magnification FR.

On the other hand, when only the lower layer data is decoded in the case where the second spatial scalability is performed, an image is reconstructed in the same manner as in the case where the first spatial scalability is performed. .

Further, when the third spatial scalability (FIGS. 7 and 8) is performed (for each object constituting the input VOP, the entire object is set as an upper layer and the entire object is thinned out). In this case, the image is reconstructed in the same manner as when the above-described second spatial scalability is performed.

As described above, the offset data FPO
S_B and FPOS_E are such that the corresponding pixels constituting the enlarged image of the lower layer and the image of the upper layer are arranged at the same position in the absolute coordinate system. With this configuration, an accurate (no displacement) image can be obtained.

Next, regarding syntax in scalability, for example, MPEG4VM (Verification M
odel) will be described as an example.

FIG. 21 shows the configuration of a bit stream obtained by scalability encoding.

The bit stream is VS (Video Sessio)
n Class), and each VS is composed of one or more VOs (Video Object Classes). And
VO is one or more VOL (Video Object Layer Class)
(When the image is not hierarchized, it is composed of one VOL, and when the image is hierarchized, it is composed of the VOL of the number of layers), and the VOL is composed of the VOL.

FIG. 22 or FIG. 23 shows the syntax of VS or VO, respectively. A VO is a bit stream corresponding to a sequence of an entire image or a part (object) of an image, and thus a VS is composed of a set of such sequences (so that a VS is, for example, a single program). Equivalent).

FIG. 24 shows the syntax of a VOL.

[0209] VOL is a class for scalability, and video_object_layer_id (in FIG. 24,
A1)).
That is, for example, video_ob for the VOL of the lower layer
ject_layer_id is set to 0, and for example, video_object_layer_id for the VOL of the upper layer is set to 1. Note that, as described above, the number of scalable layers is not limited to two, but may be any number of three or more.

[0210] For each VOL, whether it is an entire image or a part of the image is determined by video_object_l.
It is identified by ayer_shape (portion indicated by A2 in FIG. 24). The video_object_layer_shape is a flag indicating the shape of the VOL, and is set, for example, as follows.

That is, when the VOL has a rectangular shape, video_object_layer_shape is set to “00”, for example. When the VOL has a shape of an area extracted by a hard key (a binary (Binary) signal having one of 0 or 1), video_ob
ject_layer_shape is, for example, “01”. Further, when the VOL is in the shape of an area extracted by a soft key (a signal capable of taking a continuous value (Gray-Scale) in the range of 0 to 1) (combined using the soft key) Video_object_layer)
_shape is, for example, “10”.

Here, video_object_layer_shape is set to “0”.
It is set to "0" when the VOL has a rectangular shape and the position and size of the VOL in the absolute coordinate form do not change with time, that is, are constant. In this case, the sizes (horizontal length and vertical length) are video_object_layer_width and video_object_lay
er_height (portion indicated by A7 in FIG. 24). video_object_layer_width and video_ob
The ject_layer_height is a 10-bit fixed-length flag. When video_object_layer_shape is “00”, it is transmitted only once first (this is video_ob
When ject_layer_shape is “00”, as described above,
This is because the size of the VOL in the absolute coordinate system is constant.)

Further, whether the VOL is a lower layer or an upper layer is indicated by scalability (portion indicated by A3 in FIG. 24) which is a 1-bit flag. When VOL is a lower layer, scalabil
ity is, for example, 1; otherwise, scalability
Is set to 0, for example.

Further, when the VOL uses an image in a VOL other than itself as a reference image, the VOL to which the reference image belongs is, as described above, ref_layer_id (FIG. 2).
4, the portion indicated by A4). Note that ref_
layer_id is transmitted only for the upper layer.

Also, hor_samp indicated by A5 in FIG.
ling_factor_n and hor_sampling_factor_m indicate a value corresponding to the horizontal length of the VOP of the lower layer and a value corresponding to the horizontal length of the VOP of the upper layer, respectively. Therefore, the horizontal length (horizontal resolution magnification) of the upper layer with respect to the lower layer is expressed by the formula hor_sampling.
Given as _factor_n / hor_sampling_factor_m.

Further, ver_sa indicated by A6 in FIG.
mpling_factor_n and ver_sampling_factor_m indicate a value corresponding to the vertical length of the lower layer VOP and a value corresponding to the vertical length of the upper layer VOP, respectively. Therefore, the vertical length (magnification of the vertical resolution) of the upper layer with respect to the lower layer is calculated by the expression ver_sampling.
Given as _factor_n / ver_sampling_factor_m.

FIG. 25 shows a VOP (Video Object P).
lane Class).

The VOP size (horizontal and vertical lengths) is, for example, VOP_width and VOP_height (FIG. 2) having a fixed length of 10 bits.
5, the portion indicated by B1). Also, VO
The positions of P in the absolute coordinate system are, for example, VOP_horizontal_spatial_mc_ref (portion indicated by B2 in FIG. 25) and VOP_vertical_mc_ref (FIG. 25) having a fixed length of 10 bits.
, The portion indicated by B3). VOP_wi
dth or VOP_height represents the length of the VOP in the horizontal or vertical direction, respectively, and corresponds to the size data FSZ_B or FSZ_E described above. Also, VOP_ho
rizontal_spatial_mc_ref or VOP_vertical_mc_ref
Represents the horizontal or vertical coordinates (x or y coordinates) of the VOP, respectively, and corresponds to the above-described offset data FPOS_B and FPOS_E.

VOP_width, VOP_height, VOP_horizontal_
spatial_mc_ref and VOP_vertical_mc_ref are video
It is transmitted only when _object_layer_shape is other than “00”. That is, when the video_object_layer_shape is “00”, as described above, since the size and position of the VOP are all constant, the VOP_width, VOP_height, and VOP_h
orizontal_spatial_mc_ref, and VOP_vertical_mc_re
f need not be transmitted. In this case, on the receiving side, VO
P is arranged such that its upper left vertex coincides with, for example, the origin of the absolute coordinate system.
Video_object_layer_width and video_obje described in
Recognized from ct_layer_height.

Ref_select_cod indicated by B4 in FIG.
e represents an image used as a reference image as described in FIG. 15, and is specified in the VOP syntax as shown in FIG.

FIG. 26 shows a VOP (Video Object P).
lane Class) is shown.

In this embodiment, as in the case of FIG. 25, information on the size and position of the VOP is transmitted when the video_object_layer_shape is other than “00”.

However, in this embodiment, video_object
_layer_shape is other than "00", VO to be transmitted this time
26. A 1-bit flag load_VOP_size (FIG. 26) indicating whether the size of P is equal to the size of the previously transmitted VOP.
, The portion indicated by C1) is transmitted. This load_V
OP_size is set to 0 or 1, for example, when the size of the current VOP is equal to or not equal to the size of the VOP decoded immediately before.

When load_VOP_size is 0, VOP_w
idth, VOP_height (part indicated by C2 in FIG. 26)
Is not transmitted, and only when load_VOP_size is 1,
OP_width and VOP_height are transmitted. Where VOP_widt
h and VOP_height are the same as those described in FIG.

In FIGS. 25 and 26, VOP_widt
As h or VOP_height, a difference between the horizontal or vertical length of the current VOP and the horizontal or vertical length of the VOP to be decoded immediately before (hereinafter, appropriately referred to as a size difference), respectively Can be used.

In an actual image, the frequency at which the size of a VOP changes is not so high. Therefore, only when load_VOP_size is 1, VOP_width and VOP_height are transmitted to reduce redundant bits. It becomes possible. When the size difference is used, the amount of information can be further reduced.

When the size difference is used, the calculation is performed in the VLC unit 36 shown in FIGS. 11 and 12, and the VLC unit 36 converts the size difference into, for example, a variable-length code and outputs the result. Is done. In this case, the IVLC unit 102 in FIGS. 19 and 20 recognizes the size of the VOP to be decoded this time by adding the size difference and the size of the VOP decoded immediately before.

On the other hand, the information regarding the position of the VOP is not the coordinates themselves in the absolute coordinate system, but the coordinates of the current VOP and the VOP decoded immediately before (the previous VOP).
A difference value from the coordinates of P) (hereinafter, referred to as a position difference as appropriate)
Is diff_VOP_horizontal_ref, diff_VOP_vertical_ref
(The portion indicated by C3 in FIG. 26).

Here, the x or y coordinate of the immediately preceding VOP in the absolute coordinate system is represented by VOP_horizontal_mc_spat
ial_ref_prev or VOP_vertical_mc_spatial_ref_prev
Is expressed as diff_VOP_horizontal_ref or diff_VOP_
The vertical_ref is the VOP_hirizontal_mc_spat shown in FIG. 25 in the VLC unit 36 in FIGS.
ial_ref or VOP_vertical_mc_spatial_ref is calculated according to the following equation.

[0230] diff_VOP_horizontal_ref = VOP_hirizontal
_mc_spatial_ref -VOP_horizontal_mc_spatial_ref_prev diff_VOP_vertical_ref = VOP_vertical_mc_spatial_ref -VOP_vertical_mc_spatial_ref_prev

The VLC shown in FIGS. 11 and 12
In the device 36, diff_VOP_horizontal_ref, diff_VOP_vert
ical_ref is output after being subjected to variable-length coding.

That is, in the VLC unit 36, first, diff_VOP
diff_s arranged at the position indicated by C4 in FIG. 26 corresponding to _horizontal_ref or diff_VOP_vertical_ref
ize_horizontal or diff_size_vertical is obtained according to the table shown in FIG.
e) respectively. Further, in the VLC unit 36, diff_VOP_horizontal_ref or diff_VOP_vertical
_ref is diff_size_horizontal or diff_size_vertic
Al is converted into a variable-length code (Code) according to the table shown in FIG. Then, the diff_VOP_horizontal_re converted to the variable length code
f, diff_VOP_vertical_ref, diff_size_horizontal, and diff_size_vertical are multiplexed with other data and transmitted.

In this case, the IVLC shown in FIGS.
In the device 102, diff_size_horizontal or diff_size_
From vertical, diff_VOP_horizontal_ref or diff_VO
The length of the variable-length code of P_vertical_ref is recognized, and each variable-length code is decoded based on the recognition result.

As described above, in transmitting the position difference, the amount of information can be reduced as compared with the case in FIG.

The ref_sele shown by C5 in FIG.
ct_code is the same as that described with reference to FIG.

Next, FIG. 29 shows the syntax of a macroblock.

First, FIG. 29A shows the syntax of a macroblock constituting an I or P picture (VOP). A flag COD placed after the first_MMR_code at the beginning is placed after that COD. Indicates whether there is data available.

That is, the VLCs constituting lower layer coding section 25 in FIG. 11 and upper layer coding section 23 in FIG.
When the DCT coefficients (quantization results of the DCT coefficients) obtained for the macroblocks constituting the I or P picture are all 0 and the motion vector is 0, the macroblock of the I or P picture Is a skip macro block, and in this case, COD is 1. Therefore, when COD is 1, there is no data to be transmitted for the macroblock, and data after the flag COD is not transmitted.

On the other hand, when an AC component other than 0 exists in the DCT coefficient of the I or P picture, the VLC unit 36 sets the flag COD to 0, and the subsequent data is transmitted as necessary. The MCBPC arranged after the flag COD indicates the type of the macroblock, and necessary data following the MCBPC is transmitted according to the MCBPC.

Here, there is basically no case where an I picture is a skipped macroblock, so that the COD for the I picture is not transmitted (it can be prevented from being transmitted).

Next, FIG. 29 (B) shows a B picture (VO
P) indicates the syntax of the macroblocks constituting P), and the flag MODB arranged after the first first_MMR_code is the flag CDB in FIG.
It indicates whether there is data arranged after the MODB corresponding to the OD, that is, indicates the type of macroblock of the B picture.

In the VLC unit 36 shown in FIGS. 11 and 12,
The MODB is, for example, encoded into a variable length code as shown in FIG. 30 and transmitted.

That is, in the present embodiment, a table for variable-length encoding of the MODB (the table for variable-length encoding is also used for variable-length decoding of the variable-length code. 30 (A) and 30 (A) and FIG. 30 (A) and FIG.
Two types shown in (B) are prepared. In the variable length table of FIG. 30A (hereinafter referred to as MODB table A as appropriate), three variable length codes for MODB are provided. ) In the variable length table (hereinafter referred to as MODB table B as appropriate)
Are assigned two variable length codes.

The VLC unit 36 shown in FIGS.
When the ODB table A is used, a macroblock constituting a B picture is decoded using only data (a quantization coefficient, a motion vector, and the like) of a macroblock of another frame which is decoded before the macroblock is decoded. Or the macroblock at the corresponding position of the I or P picture decoded immediately before (the I or P picture at the same position as the macroblock position of the B picture currently being processed) ) Is a skipped macroblock (when COD is 1), the macroblock of the B picture is set to a skipped macroblock, and MODB is set to 0. Then, in this case, data after the MODB, including MBTYPE and CBPB, is not transmitted.

If the DCT coefficients (quantized DCT coefficients) of a macroblock all have the same value, for example, 0, etc., but a motion vector exists for the macroblock (the motion vector is transmitted). If necessary), the MODB is set to “10”, and the subsequent MBTYPE is transmitted.

Further, if at least one DCT coefficient for a macroblock is not 0 (DCT coefficient is present) and a motion vector is present for the macroblock, MODB is set to “11”, and the following MBTYPE And CBPB are transmitted.

Here, MBTYPE indicates the prediction mode of the macroblock and data (flag) included in the macroblock, and CBPB indicates which block in the macroblock has the DCT coefficient. This is a 6-bit flag. That is, as shown in FIG. 31, the macro block is a total of six blocks of a block of 8 × 8 pixels for four luminance signals and a block of 8 × 8 pixels for color difference signals Cb and Cr. The DCT unit 34 of FIGS. 11 and 12 performs DCT processing for each block. In the VLC unit 36 of FIGS. 11 and 12, each bit of the 6-bit CBPB is divided into six blocks. It is set to 0 or 1 depending on whether a DCT coefficient exists in each case.

Specifically, the macro block 6
For example, as shown in FIG. 31,
Assuming that block numbers 1 to 6 are set, the VLC unit 36, for example, sets the N-th bit of CBPB (here, for example, the least significant bit is the first bit,
The most significant bit is the sixth bit) and the block number N
Is set to 0 when no DCT coefficient exists in the block, and set to 1 when the DCT coefficient exists. Therefore, CBPB is 0 (“00
0000 ”), it means that there is no DCT coefficient for that macroblock.

On the other hand, the VLC unit 36 shown in FIGS.
In the case where the MODB table B (FIG. 30B) is used, when the MODB table A is used, when the MODB table A is set to “10” or “11”, the MODB is “0” or “10”, respectively. Is done. Therefore, MOD
When B table B is used, no skip macroblock occurs.

Next, MBTYPE is shown in FIG. 11 and FIG.
In the second VLC unit 36, for example, it is encoded into a variable length code as shown in FIG. 32 and transmitted.

That is, in the present embodiment, a variable length table for MBTYPE is used as shown in FIG.
2 (B) are prepared, and FIG. 32 (A)
In the variable length table (hereinafter, appropriately referred to as MBTYPE table A), four variable length codes for MBTYPE, and in the variable length table of FIG. 32B (hereinafter, appropriately referred to as MBTYPE table B), Three variable length codes are assigned to MBTYPE.

The VLC unit 36 shown in FIGS.
When the BTYPE table A is used, when the prediction mode is the bidirectional prediction encoding mode (Interpolate MC + Q), MBTYPE is subjected to variable-length encoding to “01”. Then, in this case, DQUANT, MVDf, and MVDb are transmitted. Here, DQUANT represents the quantization step,
MVDf or MVDb indicates a motion vector used for forward prediction or backward prediction, respectively. Note that DQ
As a UANT, not the quantization step itself,
It is possible to use the difference between the current quantization step and the previous quantization step.

When the prediction mode is the backward prediction coding mode (Backward MC + Q), MBTYPE is variable-length coded to “001”, and DQUANT, MVD
b is transmitted.

Further, when the prediction mode is the forward prediction coding mode (Forward MC + Q), MBTYPE is variable-length coded to "0001", and DQUANT, MV
Df is transmitted.

If the prediction mode is H.264, In the direct mode (Direct coding mode) defined in H.263, MBTYPE is set to “1” and MVDB is transmitted.

Here, in the above-described case, only three types of inter-coding modes, ie, forward prediction coding mode, backward prediction coding mode, and both-light prediction mode, have been described. Four types are defined by adding the direct mode to the three types. Therefore, in the motion vector detector 32 in FIGS. 11 and 12, for the B picture, for example, the intra coding mode, the forward prediction coding mode, Among the backward prediction coding mode, the bidirectional prediction mode, and the direct mode, the one that minimizes the prediction error is set as the prediction mode. The details of the direct mode will be described later.

On the other hand, the VLC unit 36 shown in FIGS.
In the case where the MBTYPE table B (FIG. 32 (B)) is used, the MBTYPE is set to “1” when it is set to “01”, “001”, or “0001” when the MPTYPE table A is used. , "0
1 "or" 001 ". Therefore, MBTYP
When the E table B is used, the direct mode is not set as the prediction mode.

Next, the direct mode will be described with reference to FIG.

For example, now, VOP0, VOP1, VO
There are four VOPs displayed in the order of P2 and VOP3, and VOP0 and VOP3 are P pictures (P-VO
P), VOP1 and VOP2 are B pictures (BV
OP). VOP0, VOP1, VO
P2 and VOP3 are VOP0, VOP3, VOP
It is assumed that encoding / decoding is performed in the order of 1, VOP2.

Under the above conditions, for example, V
The predictive encoding in the direct mode of OP1 is performed as follows.

That is, encoding (decoding) is performed immediately before VOP1.
33, that is, VO in the embodiment of FIG.
In P3, when a motion vector of a macroblock (corresponding macroblock) located at the same position as a macroblock of VOP1 to be encoded (encoding target macroblock) is MV, in the direct mode, this motion vector From the MV and the predetermined vector MVDB, a motion vector MVF for forward predictive coding of the current macroblock and a motion vector MVB for backward predictive coding are calculated according to the following equations.

MVF = (TRB × MV) / TRD + MVDB MVB = (TRB−TRD) × MV / TRD However, the motion vector MVB is calculated by the above equation when the vector MVDB is 0 and the vector MV
If DB is not 0, the motion vector MVB is calculated according to the following equation. MVB = MVF-MV Note that TRB starts from VOP1 and is an I or P picture displayed immediately before (VOP1 in the embodiment of FIG. 33).
0), and TRD represents VOP1 in the display order.
Represents the interval between I or P pictures (VOP0 and VOP3 in the embodiment of FIG. 33) immediately before and immediately after.

The motion vector detector 32 shown in FIGS. 11 and 12 calculates the vector MV for the VOP of the B picture.
DB to various values (however, the vector MVDB
Is a vector having the same direction as the motion vector MV). For example, a prediction error generated by performing predictive coding using the motion vectors MVF and MVB obtained according to the above equation is different from an intra coding mode, a forward predictive coding. When it is smaller than any one of the mode, the backward prediction coding mode, and the bidirectional prediction mode, the direct mode is set as the prediction mode.

In the embodiment shown in FIG. 33, T
Since RB = 1 and TRD = 3, the motion vector MVF is given by MV / 3 + MVDB. The motion vector MVB is 2M when MVDB is 0.
At V / 3, when MVDB is not 0, -2MV / 3 + M
Provided in VDB.

By the way, when the prediction mode is the direct mode, the most recently encoded / decoded P is used for encoding / decoding of the current macroblock.
A motion vector MV of a corresponding macroblock in a picture (VOP3 in the embodiment of FIG. 33) is required.

However, the size and position of the VOP may change (as described above,
When the object_layer_shape is “10” or “01”), in this case, the corresponding macroblock does not always exist. Therefore, in the case of performing encoding / decoding for a VOP whose size or position changes, if the direct mode is used unconditionally, a state occurs in which processing cannot be performed.

Therefore, in the present embodiment, the direct mode is a VOP (BOP) having a macroblock to be encoded.
(VOP of the picture) is the same as the VOP of the P picture to be decoded most recently. Specifically, VOP_width and VOP_h described above
Only when the size of the VOP represented by eight does not change, the use of the direct mode is permitted.

Therefore, the MB corresponding to the direct mode
In the MBTYPE table A in which the variable-length code of TYPE is defined, the size of the VOP of the B picture having the macroblock to be coded is basically the same as the size of the VOP of the most recently decoded P picture. Used only when

The MODB table A (FIG. 30)
(A)) is defined in MPEG4, and when this MODB table A is used,
When the MODB is “0” and the re
When f_select_code is not “00”, it is defined that the prediction mode is the direct mode. Therefore, the MODB table A also basically has the size of the VOP of the B picture having the encoding-target macroblock,
It is used only when the size of the VOP of the most recently decoded P picture is the same.

As described above, the MODB tables A and MB
When TYPE table A is used, when MODB is “0” or MBTYPE is “1”, the prediction mode is the direct mode.

[0271] Note that the video_object_layer_shape is set to "0".
In the case of “0”, the size of the VOP does not change, so that the MODB table A and the MBTYPE table A are used in this case as well.

On the other hand, if the VOP size of the B picture having the macroblock to be coded is different from the VOP size of the most recently decoded P picture, the direct mode cannot be used.
Is subjected to variable-length encoding / variable-length decoding using the MBTYPE table B (FIG. 32B).

If the VOP size of the B picture having the macroblock to be coded is different from the VOP size of the most recently decoded P picture, at least MPTYPE must be transmitted, that is, MB
Since it is not necessary to transmit both TYPE and CBPB, the MODB is not the MODB table A (FIG. 30 (A)) in which it is defined that both MBTYPE and CBPB are not transmitted. Is variable-length encoded / variable-length decoded using a MODB table B (FIG. 30B) in which is not defined.

As described above, by selecting (changing) the variable length table to be used in response to a change in the size of the VOP, it is possible to reduce the amount of data obtained as a result of the encoding. Become.

That is, the MODB table A (FIG. 30)
When only (A)) is used, there are only one case where the MODB is changed to a 1-bit variable length code and two cases where the MODB is changed to a 2-bit variable length code. On the other hand, when the MODB table B (FIG. 30B) is used, the MODB is 1
There is only one case where a variable length code is used for bits, and only one case where a variable length code is used for two bits. Therefore, when using both the MODB tables A and B,
Compared to the case where only the MODB table A is used, the MO
The frequency at which the DB is changed to a 2-bit variable length code is reduced, and as a result, the data amount can be reduced.

Similarly, MBTYPE table A (FIG. 32)
According to (A)), the MBTYPE may be a variable-length code of 4 bits at the longest, but in the MBTYPE table B (FIG. 32B), the MBTYPE is 3 at the longest.
It is only a variable length code of bits. Therefore,
The amount of data can be reduced.

By the way, as described above, a plurality of MODBs
When a table or MBTYPE table is used, there is no problem with the lower layer and the upper layer whose ref_select_code is other than “00”.
The following problem occurs in the upper layer in which ef_select_code is “00”.

That is, in the upper layer, the flag ref_se for the macroblock to be processed of the B picture
When the select_code is “00”,
As shown in FIG. 34, the macro block to be processed is
I or P in the same layer (here upper layer)
In this case, a picture and an image (enlarged image) at the same time of a layer different from the layer (lower layer here) at the same time are used as reference images as necessary (FIG. 15).

On the other hand, in the direct mode, a B picture located between two I or P pictures at different times is predictively coded by using a motion vector of a P picture decoded immediately before.

Therefore, ref_select_code
Is "00", the direct mode cannot be applied, but when the MBTYPE table A is used, the direct mode may be set as the prediction mode.

Therefore, in the present embodiment, the flag ref_select_code for the macroblock to be processed of the B picture is set to “0” in the upper layer.
"0", the MBTYPE is set to the variable length encoding / coding by one of the following first or second methods:
Variable-length decoding is performed.

That is, in the first method, when the flag ref_select_code for the macroblock to be processed of the B picture is “00” in the upper layer, the MBTYPE table A is not used and the MBT table is not used.
YPE table B is used. As described above, since the MBTYPE table B does not define the direct mode, the direct mode is not set as the prediction mode in the case shown in FIG.

In the second method, the following quasi-direct mode is defined as a prediction mode similar to the direct mode, and the flag ref_sele for the macroblock to be processed of the B picture is defined in the upper layer.
If ct_code is “00”, MBTYPE
When the table A is used, not the direct mode but the quasi-direct mode is assigned to the variable-length code “1” of MBTYPE.

Here, in the quasi-direct mode,
In the case shown in FIG. 34, the forward prediction is performed by using an enlarged image obtained by enlarging the image of the lower layer (different layer) in accordance with the magnification FR as a reference image (prediction reference image).
In addition, backward prediction is performed using an image encoded (decoded) immediately before an upper layer (the same layer) as a reference image.

Further, as shown in FIG. 35, when the motion vector of the corresponding macroblock (the macroblock located at the same position as the encoding-target macroblock) in the enlarged image used as the reference image for forward prediction is MV. ,
As the motion vector MVB used for backward prediction, a vector given by the following equation is used.

MVB = MV × FR + MVDB

That is, the motion vector MV for the corresponding macroblock in the lower layer is multiplied by FR, and the result obtained by adding the vector MVDB thereto is used as the motion vector MVB for backward prediction.

In this case, the motion vector MVB is not transmitted. This is because the motion vector MVB can be obtained from the motion vector MV, the scaling factor FR, and the MVDB. Therefore, on the receiving side (decoder side),
In an upper layer, a flag ref_select_code for a macroblock to be processed of a B picture
Is “00”, and when the variable length decoding is performed using the MBTYPE table A, the motion vector MV of the macroblock whose MBTYPE is “1”
B is obtained from the motion vector MV, magnification FR, and vector MVDB for the corresponding macroblock in the lower layer.

Thus, in this case, since the motion vector MVB, which is redundant data, is not transmitted, the coding efficiency can be improved.

Next, referring to the flowcharts of FIGS. 36 and 37, a method of determining a variable length table (MODB table) used in the VLC unit 36 of FIGS. 11 and 12 and the IVLC unit 102 of FIGS. 19 and 20 will be described. Whether to use A or B, and MBTYP
A method of determining which of the E tables A and B is used) will be described.

FIG. 36 shows a method of determining the variable length table used for the lower layer.

In this case, first, in step S31, whether or not the size of the VOP has changed is determined by, for example, the video_object_layer_shape described in FIG.
VOP_width, VOP_height, or loa described in FIG.
It is determined by referring to d_VOP_size and the like. If it is determined in step S31 that the magnitude of the VOP has not changed, the process proceeds to step S32, where it is determined that the MODB table A and the MBTYPE table A are used, and the process ends. On the other hand, when it is determined in step S31 that the magnitude of the VOP has changed, the process proceeds to step S33, where it is determined that the MODB table B and the MBTYPE table B are used, and the process ends.

Next, FIG. 37 shows a method of determining the variable length table used for the upper layer.

In this case, first, in step S41, it is determined whether or not ref_select_code is “00”. In step S41, ref_select_cod
If e is determined to be “00”, that is, if the VOP of the lower layer at the same time is used as a reference image for the VOP of the upper layer to be processed, the process proceeds to step S42, where the MODB table A and the MBTYPE It is determined that the table B is to be used, and the process ends.

In this case, when the quasi-direct mode is used, instead of MBTYPE table B, M
A decision is made to use BTYPE table A.
That is, in step S42, the MBTYPE table B is selected when the first method is applied, and the MBTYPE table A is selected when the second method is applied.

On the other hand, in step S41, ref_sele
If it is determined that ct_code is not “00”, the process proceeds to step S43, and thereafter, in steps S43 to S45, the same processing as in steps S31 to S33 of FIG.
The table and the MBTYPE table are determined.

Next, referring to FIGS. 38 to 40, FIG.
The processing of the skip macroblock in the lower layer encoding unit 25 of FIG. 1 and the upper layer encoding unit 23 of FIG. 12, and the lower layer decoding unit 95 of FIG. 19 and the upper layer decoding unit 93 of FIG. 20 will be described.

[0298] Here, as described above, it is basically assumed that the macroblock of the I picture is not a skipped macroblock, and therefore, the description will be given of P and B pictures.

Also, when the MODB table B is used, the skipped macroblock does not occur as described above. Therefore, the processing of the skipped macroblock is performed only when the MODB table A is used.

First, FIG. 38 is a flow chart for explaining the processing of the skipped macroblock in the lower layer coding section 25 in FIG. 11 and the lower layer decoding section 95 in FIG.

First, in step S1, it is determined whether the macroblock to be processed is a P or B picture. If it is determined in step S1 that the macroblock to be processed is a P-picture, the process proceeds to step S2, where it is determined whether the COD of the macroblock is 1. If it is determined in step S2 that the COD of the macroblock to be processed is 1, the process proceeds to step S3, where the macroblock is determined to be a skipped macroblock, and is treated as such. That is, in this case, the quantization coefficients (DCT coefficients) of the macroblock to be processed are all zero, and the motion vectors are also zero.

If it is determined in step S2 that the COD of the macroblock to be processed is not 1, the process proceeds to step S4, where the macroblock is
Usually processed. That is, in this case, the macroblock of the P picture is treated as having a DCT coefficient other than 0 or having a motion vector other than 0.

On the other hand, if it is determined in step S1 that the macroblock to be processed is a B picture,
Proceeding to step S5, it is determined whether the COD of the macroblock (corresponding macroblock) at the same position is 1 in the I or P picture decoded immediately before the macroblock of the B picture is decoded. If it is determined in step S5 that the COD of the corresponding macroblock for the macroblock to be processed is 1, the process proceeds to step S6, where the macroblock to be processed is determined to be a skip macroblock. Will be dealt with.

That is, as an image (VOP) to be processed, for example, as shown in FIG.
/ P means I or P picture), B, I / P
Are displayed in the sequence
In FIG. 3A, the leftmost I / P and the rightmost I / P
It is assumed that encoding / decoding is performed in the order of P and the second B from the left. Further, it is assumed that the macroblock of the second B picture from the left is to be processed.

In this case, the rightmost I / P picture is encoded / coded using the leftmost I / P picture as a reference image.
Will be decrypted. Therefore, when the COD of the corresponding macroblock of the rightmost I / P picture of the macroblock of the B picture to be processed is 1, that is, when the corresponding macroblock is a skip macroblock, the leftmost I / P picture is skipped. / P picture to rightmost I /
Until the P picture, there is no change in the image. Therefore, as described above, when the macroblock to be processed is a B picture and the COD of the corresponding macroblock is 1, the macroblock to be processed is set as a skipped macroblock.

In this case, the processing (prediction encoding / decoding) for the macroblock to be processed for the B picture is performed in the same manner as the corresponding macroblock for the rightmost I / P picture. The motion vector is treated as 0, and the DCT coefficients are all treated as 0 (as described above, only the MODB is transmitted on the encoder side, and the subsequent CBPB, MBTYPE, etc. are not transmitted).

Referring back to FIG. 38, if it is determined in step S5 that the COD of the corresponding macroblock is not 1, the flow advances to step S7 to determine whether the MODB of the macroblock of the B picture to be processed is 0. You. If it is determined in step S7 that the MODB is 0, the process proceeds to step S8, where the macroblock to be processed is determined to be a skipped macroblock, and is treated as such.

That is, as an image to be processed (VOP), for example, as shown in FIG.
Some are displayed and encoded / decoded in the same order as in (A), and the macroblock of the second B picture from the left is processed in the same manner as in (A) of FIG. It is assumed that it is targeted.

In this case, since the COD of the corresponding macroblock of the rightmost I / P picture of the macroblock of the B picture to be processed is not 1, that is, since the corresponding macroblock is not a skip macroblock, The image has changed between the left I / P picture and the rightmost I / P picture.

[0310] On the other hand, since the MODB of the macroblock of the B picture to be processed is 0, this macroblock is decoded by using only the data of the macroblock of another frame which is decoded by the time it is decoded. The corresponding macroblock in the I or P picture decoded immediately before or can be decoded is a skip macroblock (COD is 1), but as described above, since COD is not 1, , A macroblock of a B picture to be processed can be decoded by using data of a macroblock of another frame (hereinafter, appropriately referred to as decoded data) which is decoded before the macroblock is decoded. Will be.

Therefore, there is a change in the image between the leftmost I / P picture and the rightmost I / P picture, and the macroblock of the B picture to be processed is determined using only the decoded data. Considering the case where decoding is possible, for example, as shown in FIG. 40B, the corresponding macroblock (part shown by a solid line) in the rightmost I / P picture is replaced with the leftmost I / P picture. Is processed as a reference image, the motion vector MV1 is, for example, 1
A predicted image obtained by motion-compensating the leftmost I / P picture or the rightmost I / P picture with the motion vector MV2 or MV3 multiplied by ２ or − ／, respectively (in FIG. This means that the average value of the portion indicated by the dotted line) matches the macroblock to be processed (when no prediction error occurs).

From the above, in step S8 of FIG.
The processing (prediction encoding / decoding) of the macroblock to be processed in the picture is performed by using a motion vector MV2 (M
VF) and MV3 (MVB) are used, and the average value of the above-described predicted image is used as the pixel value (image data).

That is, in this case, for example, the prediction mode for the macroblock to be processed is the direct mode described above. In addition, H. In H.263, it is the PB picture to which the direct mode is applied,
The B picture in the present embodiment refers to MPEG1, MPEG2,
B picture and H. This is a broad concept including the PB picture in H.263.

On the other hand, if it is determined in step S7 that the MODB for the macroblock of the B picture to be processed is not 0, the process proceeds to step S9, and the normal processing is performed as in step S4.

Next, FIG. 39 is a flowchart for explaining the processing of a skipped macroblock in the upper layer coding section 23 in FIG. 12 and the upper layer decoding section 93 in FIG.

In this case, in steps S11 to S14, the same processes as those in steps S1 to S4 in FIG. 38 are performed. That is, the same processing is performed on the P picture for both the lower layer and the upper layer.

On the other hand, if it is determined in step S11 that the macroblock to be processed is a B picture, the flow advances to step S15 to set the flag ref_select_code for the macroblock to "00".
Is determined. In step S15, when it is determined that the flag ref_select_code for the macroblock of the B picture to be processed is not “00”, that is, the macroblock of the B picture uses a lower layer image at the same time as a reference image as a reference image. If not, the process proceeds to steps S16 to S20 and proceeds to steps S5 to S in FIG.
9, the same processing is performed.

In step S15, the flag ref for the macroblock of the B picture to be processed is
If it is determined that _select_code is “00”, that is, if the macroblock of the B picture is to be processed using the lower-layer image at the same time as the reference image, the process proceeds to step S21.
MO for macroblock of B picture to be processed
It is determined whether DB is 0.

If it is determined in step S21 that the MODB for the macroblock of the B picture to be processed is 0, the process proceeds to step S22, where the macroblock to be processed is determined to be a skip macroblock, Treated that way. If it is determined in step S21 that the MODB of the macroblock of the B picture to be processed is not 0, the process proceeds to step S23, and normal processing is performed as in step S3 of FIG.

That is, as the upper layer image (VOP) to be processed, for example, as shown in FIG. 40 (C), an image displayed in a sequence of I / P, B, B,. It is assumed that there are some lower layer images that are displayed in the same sequence. Further, it is assumed that these lower layer images and upper layer images are alternately encoded / decoded. Note that ref_select_c for the B picture in the upper layer
When the mode is “00”, the encoding / decoding order of the image is as follows.

In this case, ref_select
If the value of _code is not determined in step S15, that is, if the same processing as described with reference to FIG. 38 is performed, the macroblock of the B picture in the upper layer to be processed is shown in FIG. As described above, encoding / decoding is performed using an image (enlarged image) of the lower layer at the same time or an immediately preceding decoded image (leftmost I / P picture) of the upper layer as a reference image. The COD (or MODB) value for the corresponding macroblock in such a subsequent frame determines whether the macroblock to be processed is a skipped macroblock, even though the subsequent frame is not referenced. Become.

However, it is not preferable to determine whether a macroblock to be processed is a skipped macroblock based on a frame that is not referred to when encoding / decoding the macroblock to be processed.

Therefore, in the embodiment shown in FIG.
For the B picture of the upper layer, ref_sel
Based on ct_code, if it is "00",
That is, the macro block of the B picture is
As shown in (C), when processing is performed using an image (enlarged image) of the lower layer at the same time or an immediately preceding decoded image (leftmost I / P picture) of the upper layer as a reference image. , Regardless of the COD or MODB of the corresponding macroblock in the subsequent frames, whether or not the macroblock is a skip macroblock is determined according to the MODB of the macroblock of the B picture to be processed.

[0324] When ref_select_code is "00", the MODB of the macroblock of the B picture to be processed becomes 0 as a reference image in the upper layer, not in the image at the same time of the lower layer. The immediately preceding decoded image (the leftmost I
/ P picture), the processing (prediction encoding / decoding) for the macroblock to be processed is performed using the immediately preceding decoded image as a reference image and the motion vector as Performed as 0.

The processing of the skip macro block has been described above. In the case of performing such processing,
Whether the macroblock to be processed is one of the upper layer and the lower layer is determined based on the flag scalability described with reference to FIG.

Here, the motion vector detector 3 shown in FIG.
The reason why the COD of the lower layer is supplied to the VLC unit 36 and the motion compensator 42 is as follows.

That is, for example, in the case of the temporal scalability shown in FIG. 14, as described above, the image of the lower layer is used as the reference image for the prediction of the upper layer. In this case, for example, VOP0 of the lower layer, VOP of the upper layer
OP1 and the lower layer VOP2 are images that are temporally continuous, and therefore, these three VOP1 and VOP
2. If VOP3 satisfies the condition described with reference to FIG. 40A, the macroblock of VOP1 in the upper layer is a skipped macroblock. If the macroblock is a skip macroblock, the macroblock does not need to be processed. On the other hand, whether or not the condition described with reference to FIG.
Since the COD of OP2 is necessary, the motion vector detector 32 in the upper layer coding unit 23 in FIG.
The lower layer COD is supplied to the VLC unit 36 and the motion compensator 42.

Next, in MPEG4, except for the case where the prediction mode is direct, the DC
Although it is specified that DQUANT for the quantization step should be transmitted even when all the T coefficients have a predetermined value such as 0 due to quantization (when no DCT coefficient exists), for example. , DCT of macroblock
Transmitting DQUANT when no coefficients are present is redundant.

Therefore, the VLC unit 3 shown in FIGS.
6, and the IVLC unit 102 in FIGS. 19 and 20, the quantization step DQUANT is handled as follows.

That is, first, in step S51, it is determined whether or not CBPB is 0.
Is determined to be 0, the DCT of the macroblock
Since there is no coefficient, the process proceeds to step S56, and the quantization step is ignored (the quantization step DQUANT is not transmitted on the encoder side, and the quantization step DQUANT is not extracted from the bit stream on the decoder side (the Is not possible)), and the processing ends.

Here, as described with reference to FIG.
B may not be transmitted, but in this case, the process of step S51 is skipped, and the process of step S52 is performed.

On the other hand, in step S51, CBPB
If it is determined that is not 0, the process proceeds to step S52,
It is determined whether or not MODB is 0. Step S52
, When the MODB is determined to be “0”,
As described with reference to FIG. 30, since CBPB is not transmitted, and therefore, no DCT coefficient of the macroblock exists, the process proceeds to step S56, the quantization step is ignored, and the process ends.

Also, in step S52, MODB
Is not “0”, the process proceeds to step S53, and either of the MODB tables A or B
It is determined whether ODB is used for variable-length encoding / variable-length decoding of ODB. If it is determined in step S53 that the MODB table B is to be used, the process proceeds to step S54.
And skips to step S55. If it is determined in step S53 that the MODB table A is to be used, the process proceeds to step S54, where it is determined whether the MODB is "10".

In step S54, MODB is set to "1".
If it is determined to be “0”, that is, if the MODB table A is used and the MODB is “10”, CBPB is not transmitted as described with reference to FIG. Since there is no DCT coefficient of the block, the process proceeds to step S56, the quantization step is ignored, and the process ends.

On the other hand, in step S54,
Is determined to be not “10”, a process for the quantization step is performed (the quantization step DQUANT is transmitted on the encoder side, and the quantization step DQUANT is extracted from the bit stream on the decoder side), The process ends.

As described above, when the DCT coefficient of the macroblock does not exist, that is, when CBPB is 0, the MO
MODB is "0" when using DB table A
Alternatively, when the value is "10", and when MODB is "0" in the case where the MODB table B is used, the quantization step is ignored, so that the data redundancy can be reduced.

When CBPB is transmitted and the value is "0", the case where the MODB is set to "11" or "10" using the MODB table A or B, respectively. , In such a case, MODB
Use "10" or "0" respectively.
Basically not. Accordingly, in the embodiment of FIG. 41, the value of CBPB is determined in the first step S51, but this determination processing is performed immediately before the processing of step S55 from the viewpoint of processing efficiency. It is desirable.

The processing shown in FIG. 41 is applicable to any of the first and second methods.

As described above, the V for which the position and size are converted
O is arranged and processed in the absolute coordinate system.
Predictive encoding / decoding for each VO becomes possible.
Scalability for O can be realized.

Further, the processing of the skipped macroblock is determined in consideration of the flag ref_select_code indicating the reference image used for the skipped macroblock, so that efficient processing can be performed.

In the case where the upper layer and the lower layer have the same image, when a lower layer decoded image at the same time is used as a reference image for predictive coding of the upper layer, the motion vector in the upper layer is Since only the data in the lower layer is transmitted without being transmitted, the data amount can be reduced.

The processing described in this embodiment as being performed in units of macroblocks can be performed in units other than units of macroblocks.

In this embodiment, two types of MODs are used.
Although B tables A and B are prepared and one of them is selected and used, it is also possible to prepare three or more types of MODB tables. This means that MBTYPE
The same applies to the table.

[0344]

[Effect of the Invention] The image coding method according to the image coding apparatus and claim 2 of claim 1, as well as the picture decoding method according to the image decoding apparatus and claim 4 of claim 3 According to this, when a macroblock of a B picture is coded by one of forward prediction coding, backward prediction coding, and bidirectional prediction coding, the macroblock
I or P picture encoded just before is encoded
Of the B picture of the macroblocks
The one corresponding to the local block is the skip macro block.
At one time, a B picture is a skip macroblock
It is determined that there is. Therefore, if the macroblock is encoded
Construct an I or P picture encoded just before
Macroblock of B picture among macroblocks
It is possible to prevent a process that is not performed from being performed as a skipped macroblock .

[0345] According to the recording method according to claim 5, forward predictive coding, backward predictive coding, or by any of the bidirectional predictive coding, when encoding a macroblock of a B picture, a macro block Just before is encoded
Macro block that constitutes an I or P picture encoded in
Of the lock, corresponding to the macroblock of the B picture
When the object is a skip macro block,
Cha is determined to be a skip macroblock.
Therefore, just before the macroblock is coded,
Macroblocks that make up the I or P picture
The one that does not correspond to the macroblock of the B picture
It is possible to prevent the processing from being performed as a skipped macroblock .

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of an encoder to which the present invention is applied.

FIG. 2 is a diagram for explaining that a position and a size of a VO change with time.

FIG. 3 is a block diagram illustrating a configuration example of VOP encoding units 31 to 3N in FIG. 1;

FIG. 4 is a block diagram illustrating another configuration example of the VOP encoding units 31 to 3N of FIG. 1;

FIG. 5 is a diagram for explaining spatial scalability.

FIG. 6 is a diagram for describing spatial scalability.

FIG. 7 is a diagram for describing spatial scalability.

FIG. 8 is a diagram for describing spatial scalability.

FIG. 9 is a diagram for explaining a method for determining VOP size data and offset data.

FIG. 10 is a diagram for explaining a method of determining VOP size data and offset data.

11 is a block diagram illustrating a configuration example of a lower layer encoding unit 25 in FIG.

12 is a block diagram illustrating a configuration example of an upper layer encoding unit 23 in FIG.

FIG. 13 is a diagram for describing spatial scalability.

FIG. 14 is a diagram for explaining time scalability.

FIG. 15 shows a reference select code (ref_sele
FIG. 3 is a diagram for explaining (ct_code).

FIG. 16 is a block diagram illustrating a configuration of an embodiment of a decoder to which the present invention has been applied.

17 is a block diagram illustrating a configuration example of VOP decoding units 721 to 72N of FIG.

18 is a block diagram illustrating another configuration example of the VOP decoding units 721 to 72N of FIG.

19 is a block diagram illustrating a configuration example of a lower layer decoding unit 95 in FIG.

20 is a block diagram illustrating a configuration example of an upper layer decoding unit 93 in FIG.

FIG. 21 is a diagram illustrating the syntax of a bit stream obtained by scalable encoding.

FIG. 22 is a diagram illustrating the syntax of a VS.

FIG. 23 is a diagram illustrating the syntax of a VO.

FIG. 24 is a diagram illustrating the syntax of a VOL.

FIG. 25 is a diagram illustrating VOP syntax.

FIG. 26 is a diagram illustrating the syntax of a VOP.

FIG. 27: diff_size_horizontal and diff_size_vert
It is a figure showing the variable length code of ical.

FIG. 28: diff_VOP_horizontal_ref and diff_VOP_ve
It is a figure showing the variable length code of rtical_ref.

FIG. 29 is a diagram illustrating the syntax of a macroblock.

FIG. 30 is a diagram illustrating a variable-length code of a MODB.

FIG. 31 is a diagram illustrating a configuration example of a macroblock.

FIG. 32 is a diagram showing a variable length code of MBTYPE.

FIG. 33 is a diagram for describing predictive encoding in a direct mode.

FIG. 34 is a diagram for describing predictive encoding of a B picture in an upper layer.

FIG. 35 is a diagram for explaining a quasi-direct mode.

FIG. 36 is a flowchart illustrating a method for determining a variable length table used for a lower layer.

FIG. 37 is a flowchart illustrating a method for determining a variable length table used for an upper layer.

FIG. 38 is a flowchart illustrating a process for a skip macroblock in a lower layer.

FIG. 39 is a flowchart illustrating a process for a skipped macroblock in an upper layer.

FIG. 40 is a diagram for describing processing for a skipped macroblock.

FIG. 41 is a flowchart illustrating a process of a quantization step DQUANT.

FIG. 42 is a block diagram illustrating a configuration of an example of a conventional encoder.

FIG. 43 is a block diagram showing a configuration of an example of a conventional decoder.

FIG. 44 is a block diagram illustrating a configuration of an example of a conventional encoder that performs scalable encoding.

FIG. 45 is a block diagram illustrating a configuration example of a lower layer encoding unit 202 in FIG. 44.

FIG. 46 is a block diagram illustrating a configuration example of an upper layer encoding unit 201 in FIG. 44.

FIG. 47 is a block diagram illustrating a configuration of an example of a conventional decoder that performs scalable decoding.

FIG. 48 is a block diagram illustrating a configuration example of a lower layer decoding unit 232 in FIG. 47.

FIG. 49 is a block diagram illustrating a configuration example of an upper layer decoding unit 231 in FIG. 47.

FIG. 50 is a diagram for explaining a conventional image synthesizing method.

FIG. 51 is a diagram for describing an encoding method that enables re-editing and re-synthesis of an image.

FIG. 52 is a diagram for describing a decoding method that enables re-editing and re-synthesis of an image.

[Explanation of symbols]

1 VO component, 21 to 2N VOP component, 31
To 3N VOP encoding unit, 4 multiplexing unit, 21
Image layering unit, 23 upper layer encoding unit, 24 resolution conversion unit, 25 lower layer encoding unit, 26 multiplexing unit, 31 frame memory, 32 motion vector detector, 33 arithmetic unit, 34 DCT unit, 35
Quantizer, 36 VLC unit, 38 inverse quantizer,
39 IDCT unit, 40 arithmetic unit, 41 frame memory, 42 motion compensator, 43 key signal encoding unit,
44 key signal decoder, 51 key signal encoder,
52 key signal decoding unit, 53 frame memory,
71 demultiplexer, 721 to 72N VOP decoder,
73 image reconstruction unit, 91 demultiplexing unit, 93 upper layer decoding unit, 94 resolution conversion unit, 95 lower layer decoding unit, IVLC unit, 103 dequantizer,
104 IDCT unit, 105 arithmetic unit, 106 frame memory, 107 motion compensator, 108, 11
1-key signal decoder, 112 frame memory

Claims (5)

(57) [Claims] 1. An I-picture for intra-coding an image of two or more layers, a P-picture for intra-coding or forward prediction coding, and intra-coding, forward prediction coding, backward prediction coding, or both directions An image coding apparatus that classifies into a B picture to be predictively coded and performs coding in units of macroblocks, wherein said B picture is encoded by one of forward prediction coding, backward prediction coding, and bidirectional prediction coding. when encoding macroblocks in the picture, the macroblock code
Constructs an I or P picture coded immediately before being encoded
Macro of the B picture,
The one corresponding to the block is the skip macro block.
When the B picture is a skip macro block
An image encoding device characterized in that: 2. An I-picture for intra-coding an image of two or more layers, a P-picture for intra-coding or forward predictive coding, and intra-coding, forward predictive coding, backward predictive coding, or both directions An image coding method of classifying into a B picture to be predictively coded and coding in units of macroblocks, wherein the B picture is encoded by one of forward prediction coding, backward prediction coding, and bidirectional prediction coding. when encoding macroblocks in the picture, the macroblock code
Constructs an I or P picture coded immediately before being encoded
Macro of the B picture,
The one corresponding to the block is the skip macro block.
When the B picture is a skip macro block
An image encoding method characterized by determining that 3. An I-picture for intra-coding an image of two or more layers, a P-picture for intra-coding or forward predictive coding, and intra-coding, forward predictive coding, backward predictive coding, or both directions An image decoding apparatus that classifies a B picture to be predictively coded and decodes coded data coded in macroblock units, comprising: forward prediction coding, backward prediction coding, or bidirectional prediction coding. When decoding the macroblock of the B picture according to any of the above, the macroblock is decoded
A decoded I or P picture immediately before being decoded
Macro of the B picture,
The one corresponding to the block is the skip macro block.
When the B picture is a skip macro block
An image decoding device, characterized in that: 4. An I-picture for intra-coding an image of two or more layers, a P-picture for intra-coding or forward-prediction coding, and intra-coding, forward-prediction coding, backward-prediction coding, or both directions An image decoding method for classifying into B pictures to be predictively coded and decoding coded data coded in macroblock units, comprising: forward prediction coding, backward prediction coding, or bidirectional prediction coding. When decoding the macroblock of the B picture according to any of the above, the macroblock is decoded
A decoded I or P picture immediately before being decoded
Macro of the B picture,
The one corresponding to the block is the skip macro block.
When the B picture is a skip macro block
An image decoding method, characterized in that: 5. An I-picture for intra-coding an image of two or more layers, a P-picture for intra-coding or forward-prediction coding, and intra-coding, forward-prediction coding, backward-prediction coding, or both directions A recording method for recording encoded data categorized into B pictures to be predictively encoded and encoded in macroblock units, comprising one of forward prediction encoding, backward prediction encoding, and bidirectional prediction encoding Accordingly, when encoding a macroblock of the B picture, the macroblock code
Constructs an I or P picture coded immediately before being encoded
Macro of the B picture,
The one corresponding to the block is the skip macro block.
When the B picture is a skip macro block
The recording method characterized by determining that