JP3263901B2 - Image signal encoding method and apparatus, image signal decoding method and apparatus - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image signal recorded on a recording medium such as a magneto-optical disk or a magnetic tape, reproduced and displayed on a display device, a video conference system, a video telephone system, and the like. , Broadcasting equipment, multimedia database search system, etc.
When a moving image signal is transmitted from a transmission side to a reception side via a transmission path, and the reception side receives and displays the signal,
The present invention relates to an image signal encoding method and an image signal encoding device, an image signal decoding method, and an image signal decoding device suitable for use in editing and recording a moving image signal.

[0002]

2. Description of the Related Art For example, in a system for transmitting a moving image signal to a remote place, such as a video conference system and a video telephone system, in order to efficiently use a transmission path, line correlation or inter-frame correlation of a video signal is required. Is used to compress and encode an image signal.

A typical high-efficiency moving picture coding method is the so-called MPEG (moving picture coding for storage) method. This is ISO-IEC / JTC1 / SC2 /
It is discussed at WG11 and proposed as a standard. Motion compensated predictive coding and DCT (Discrete Cosine Tr
ansform) A hybrid system combining coding is adopted. In MPEG, several profiles and levels are defined to support various applications and functions. The most basic is the main profile main level (MP @ ML).

[0004] Referring to FIG. 20, the MP @
A configuration example of the ML encoder will be described.

An input image signal is first sent to a frame memory 20.
1 and thereafter read from the frame memory 201, sent to the subsequent configuration, and encoded in a predetermined order.

That is, the image data to be encoded is
The data is read from the frame memory 201 in macroblock units and input to the motion vector detection (ME) circuit 202. The motion vector detection circuit 202 processes the image data of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance. Whether the image of each frame input sequentially is processed as one of I, P, and B pictures is predetermined (for example, I, B, P,
B, P,... B, P are processed in this order).

Here, the motion vector detecting circuit 202
The motion compensation is performed by referring to a predetermined reference frame, and the motion vector is detected. Motion compensation (inter-frame prediction) has three types of prediction modes: forward prediction, backward prediction, and bidirectional prediction. The prediction mode of a P picture is only forward prediction, and the prediction mode of a B picture is of three types: forward prediction, backward prediction, and bidirectional prediction. The motion vector detection circuit 202 selects a prediction mode that minimizes a prediction error, and generates a motion vector at that time.

At this time, the prediction error is compared with, for example, the variance value of the macroblock to be coded. If the variance value of the macroblock is smaller, the prediction is not performed on that macroblock and the intra-frame coding is performed. Will be in this case,
The prediction mode is intra-picture coding (intra).

The motion vector and the information on the prediction mode are input to a variable length coding (VLC) circuit 206 and a motion compensation (MC) circuit 212.

The motion compensation circuit 212 generates a predicted reference image based on a predetermined motion vector, and inputs the predicted reference image signal to the arithmetic circuit 203. The arithmetic circuit 203 obtains a difference signal between the value of the macroblock to be encoded from the frame memory 201 and the value of the macroblock of the predicted reference image from the motion compensation circuit 212, and sends the difference signal to the DCT circuit 204. Output. In the case of an intra macroblock (a macroblock to be intra-coded), the arithmetic circuit 203 outputs the macroblock signal to be coded to the DCT circuit 204 as it is.

In the DCT circuit 204, the arithmetic circuit 20
3 is subjected to DCT (Discrete Cosine Transform) processing and converted into DCT coefficients. The DCT coefficient is input to the quantization circuit 205, where it is quantized in a quantization step corresponding to the amount of data stored in the transmission buffer 207 (the remaining amount of data that can be stored in the buffer), and then the variable length coding circuit 206 is input.

The variable length coding circuit 206 performs quantization (Q)
In accordance with the quantization step (quantization scale) supplied from the circuit 205, the image data (in this case, I-picture data) supplied from the quantization circuit 205 is converted into a variable length code such as a Huffman code. , And outputs this code to the transmission buffer 207.

The variable length coding circuit 206 also includes a quantization step (scale) from the quantization circuit 205 and a prediction mode (intra-picture prediction, forward prediction, backward prediction, Or, a prediction mode indicating which of the bidirectional predictions is set) and a motion vector are input, and these are also subjected to variable-length coding.

The transmission buffer 207 temporarily stores the input data and outputs (buffer feedback) data corresponding to the stored amount to the quantization circuit 205 as a quantization control signal. That is, the transmission buffer 207
Is output from the quantization circuit 205 by increasing the quantization scale of the quantization circuit 205 by the quantization control signal when the data storage amount (remaining amount of storable data) increases to an allowable upper limit value. The data amount of the quantized data is reduced. Conversely, when the data storage amount (remaining amount of data that can be stored) decreases to the permissible lower limit, the transmission buffer 207 reduces the quantization scale of the quantization circuit 205 by using the quantization control signal. , The amount of quantized data output from the quantization circuit 205 is increased. In this way, overflow or underflow of the transmission buffer 207 is prevented.

[0015] The data stored in the transmission buffer 207 is read at a predetermined timing and output as a bit stream to a transmission path.

On the other hand, the data output from the quantization circuit 205 is also input to an inverse quantization (IQ) circuit 208.
In the inverse quantization circuit 208, the data supplied from the quantization circuit 205 is inversely quantized according to the quantization step also supplied from the quantization circuit 205. The output of the inverse quantization circuit 208 is input to an IDCT (inverse DCT) circuit 209, where it is subjected to inverse DCT processing, and then sent to an arithmetic circuit 210. In this arithmetic circuit 210, I
An image signal is restored by adding the output of the DCT circuit 209 and the output of the motion compensation circuit 212. This image signal is stored in the frame memory 211. Motion compensation circuit 212
Generates a prediction reference image using the image in the frame memory 211, the motion vector, and the prediction mode.

Next, referring to FIG.
A configuration example of the ML decoder will be described.

The encoded image data (bit stream) transmitted via the transmission path is received by a receiving circuit (not shown) or reproduced by a reproducing device, and temporarily stored in a receiving buffer 221. It is supplied to a variable length decoding (IVLC) circuit 222. The variable length decoding circuit 222 performs variable length decoding on the data supplied from the reception buffer 221,
The obtained motion vector and the prediction mode are stored in the motion compensation circuit 22.
7, and the quantization step to the inverse quantization circuit 223,
In addition to outputting them, the decoded image data is output to an inverse quantization (IQ) circuit 223.

The inverse quantization circuit 223 includes the variable length decoding circuit 2
The image data supplied from 22 is inversely quantized according to the quantization step also supplied from the variable length decoding circuit 222, and is output to the IDCT circuit 224. Inverse quantization circuit 2
The data (DCT coefficient) output from IDT 23 is
The inverse DCT processing is performed by the circuit 224, and the result is supplied to the arithmetic circuit 225.

If the image data supplied from the IDCT circuit 224 is I-picture data, the data is output as it is from the arithmetic circuit 225, and the image data (P Alternatively, the data is supplied to and stored in the frame memory 226 in order to generate predicted reference image data of B picture data).
The output data from the arithmetic circuit 225 is directly output as a reproduced image to the outside.

On the other hand, if the input bit stream is P or B
In the case of a picture, the motion compensation circuit 227 generates a predicted reference image according to the motion vector and the prediction mode supplied from the variable length decoding circuit 222, and outputs the predicted reference image to the arithmetic circuit 225. In the arithmetic circuit 225, the ID
The image data input from the CT circuit 224 and the predicted reference image data supplied from the motion compensation circuit 227 are added to generate an output image. In the case of a P picture, the arithmetic circuit 2
The output of 25 is also input to and stored in the frame memory 226, and is used as a reference image of an image signal to be decoded next.

In MPEG, various profiles and levels are defined in addition to MP @ ML, and various tools are prepared. The scalability described below is also MP
This is one of the EG's tools.

In MPEG, a scalable encoding method for realizing scalability corresponding to different image sizes and frame rates has been introduced. For example, in the case of spatial scalability, when decoding only the lower layer bit stream, an image signal with a small image size is decoded, and when decoding the lower layer and upper layer bit streams, an image signal with a large image size is decoded. I do.

An encoder for spatial scalability will be described with reference to FIG. In the case of spatial scalability, the lower layer corresponds to an image signal with a small image size, and the upper layer corresponds to an image signal with a large image size.

The image signal of the lower layer is first input to the frame memory 261 and the above-described M
Encoded in the same way as P @ ML.

That is, the data read from the frame memory 261 in macroblock units is input to the motion vector detection circuit 262. The motion vector detection circuit 262 processes the image data of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance.

The motion vector detecting circuit 262 refers to a predetermined reference frame (ie, a front original image, a rear original image, and an original image), performs motion compensation, and detects the motion vector. Motion compensation (inter-frame prediction) has three types of prediction modes: forward prediction, backward prediction, and bidirectional prediction. The motion vector detection circuit 262 selects a prediction mode that minimizes a prediction error, and generates a motion vector at that time. The information on the motion vector and the prediction mode is stored in the variable length coding circuit 266 and the motion compensation circuit 2.
72.

The motion compensation circuit 272 generates a predicted reference image based on a predetermined motion vector, and inputs this predicted reference image signal to the arithmetic circuit 263. The arithmetic circuit 263 obtains a difference signal between the value of the macroblock to be encoded from the frame memory 261 and the value of the macroblock of the predicted reference image from the motion compensation circuit 272, and sends the difference signal to the DCT circuit 264. Output. In the case of an intra macroblock (a macroblock to be intra-coded), the arithmetic circuit 263 outputs the macroblock signal to be encoded to the DCT circuit 264 as it is.

In the DCT circuit 264, the operation circuit 26
3 is subjected to DCT processing and converted into DCT coefficients. The DCT coefficient is input to the quantization circuit 265, where it is quantized in a quantization step corresponding to the amount of data stored in the transmission buffer 267 (the remaining amount of data that can be stored in the buffer), and then the variable length coding circuit 266.

The variable length coding circuit 266 is
Quantization step (quantization scale) supplied from 65
, The image data supplied from the quantization circuit 265 is converted into a variable-length code such as a Huffman code,
This code is output to the transmission buffer 267.

The variable length coding circuit 266 also includes a quantization step (scale) from the quantization circuit 265 and a prediction mode (intra-picture prediction, forward prediction, backward prediction, Or, a prediction mode indicating which of the bidirectional predictions is set) and a motion vector are input, and these are also subjected to variable-length coding.

The transmission buffer 267 temporarily stores the input data, and outputs (buffer feedback) data corresponding to the storage amount to the quantization circuit 265 as a quantization control signal. Thereby, the transmission buffer 2
The overflow or underflow of 67 is prevented.

The data stored in the transmission buffer 267 is read out at a predetermined timing and output to a transmission path as a bit stream.

On the other hand, the data output from the quantization circuit 265 is also input to the inverse quantization circuit 268. In the inverse quantization circuit 268, the data supplied from the quantization circuit 265 is inversely quantized according to the quantization step also supplied from the quantization circuit 265. The output of the inverse quantization circuit 268 is input to the IDCT circuit 269,
Here, after the inverse DCT processing, the signal is sent to the arithmetic circuit 270. The arithmetic circuit 270 restores an image signal by adding the output of the IDCT circuit 269 and the output of the motion compensation circuit 272. This image signal is stored in the frame memory 271. The motion compensation circuit 272 includes the frame memory 27
A prediction reference image is generated using one image, a motion vector, and a prediction mode.

However, in the configuration example of the lower layer, the output of the arithmetic circuit 270 is
1 is not only used as the reference image of the lower layer but also used as a reference image of the lower layer, and after being enlarged to the same image size as the image size of the upper layer by the image enlargement circuit 243 that enlarges the image by upsampling, It is also used for

That is, the output of the arithmetic circuit 270 is input to the frame memory 271 and the image enlargement circuit 243 as described above. The image enlargement circuit 243 enlarges the image signal generated by the arithmetic circuit 270 to have the same size as the image size of the upper layer and outputs the same to the weighting circuit 244.

In the weighting circuit 244, the image enlargement circuit 2
43 is multiplied by a weight (1-W), and an arithmetic circuit 258
Output to

On the other hand, the image signal of the upper layer is first input to the frame memory 245. The motion vector detection circuit 246 determines a motion vector and a prediction mode, similarly to the above-described MP @ ML.

Here, in the configuration of the upper layer,
The motion compensation circuit 256 generates a prediction reference image according to the motion vector determined by the motion vector detection circuit 246 and the prediction mode. The prediction reference image signal is supplied to the weighting circuit 257. The weighting circuit 257 multiplies the predicted reference image by a weight W (weight count W) and outputs the result to the arithmetic circuit 258.

The arithmetic circuit 258 adds the outputs of the weighting circuits 244 and 257, and outputs the obtained image signal to the arithmetic circuit 247 as a predicted reference image. Arithmetic circuit 25
8 is also input to the arithmetic circuit 254 and the inverse DCT
After being added to the output of the circuit 253, the frame memory 25
5 and is used as a reference image of an image signal to be encoded thereafter.

The arithmetic circuit 247 includes a frame memory 245
From the image signal to be encoded and the output of the arithmetic circuit 258, and outputs the calculated difference. However, in the case of the intra-coded macroblock, the arithmetic circuit 247 outputs the image signal to be encoded to the DCT circuit 248 as it is.

The DCT circuit 248 performs a DCT (Discrete Cosine Transform) process on the output of the arithmetic circuit 247, generates DCT coefficients, and outputs the DCT coefficients to the quantization circuit 249. Quantization circuit 249
In the same manner as in the case of MP ＠ ML, the DCT coefficients are quantized according to the quantization scale determined based on the data storage amount of the transmission buffer 251 and the like, and the variable length coding circuit 2
Output to 50. The variable-length coding circuit 250 performs variable-length coding on the quantized DCT coefficients,
The bit stream of the upper layer is output via 51.

The output of the quantization circuit 249 is inversely quantized by the inverse quantization circuit 252 using the quantization scale used in the quantization circuit 249, and
After being dequantized by 3, the signal is input to the arithmetic circuit 254.
In the arithmetic circuit 254, the arithmetic circuit 258 and the inverse DCT circuit 2
The outputs of 53 are added, and the obtained image signal is input to the frame memory 255.

The variable length coding circuit 250 also includes the motion vector and prediction mode detected by the motion vector detection circuit 246, the quantization scale used by the quantization circuit 249, and the weights used by the weighting circuits 244 and 257. W
Are input, respectively encoded and transmitted.

Next, an example of a spatial scalability decoder will be described with reference to FIG.

After the bit stream of the lower layer is input to the reception buffer 301, it is decoded like MP @ ML. That is, the data read from the reception buffer 301 is sent to the variable length decoding circuit 302. The variable length decoding circuit 302 performs variable length decoding on the data supplied from the reception buffer 301, and outputs the motion vector and the prediction mode to the motion compensation circuit 307, and outputs the quantization step to the inverse quantization circuit 303, respectively. , And outputs the decoded image data to the inverse quantization circuit 303.

The inverse quantization circuit 303 includes a variable length decoding circuit 3
02 is inversely quantized according to the quantization step also supplied from the variable length decoding circuit 302, and is output to the IDCT circuit 304. Inverse quantization circuit 3
03 (DCT coefficient) is the IDCT
The circuit 304 performs an inverse DCT process and supplies the result to the arithmetic circuit 305.

If the image data supplied from the IDCT circuit 304 is I-picture data, the data is output from the arithmetic circuit 305 and the image data (P or B-picture The data is supplied to and stored in the frame memory 306 to generate predicted reference image data of the data. This data is output to the outside as a reproduced image as it is.

On the other hand, if the input bit stream is P or B
In the case of a picture, the motion compensation circuit 307 generates a predicted reference image according to the motion vector and the prediction mode supplied from the variable length decoding circuit 302, and outputs the predicted reference image to the arithmetic circuit 305. In the arithmetic circuit 305, the ID
The image data input from the CT circuit 304 and the predicted reference image data supplied from the motion compensation circuit 307 are added to generate an output image. In the case of a P picture, the output of the arithmetic circuit 305 is also input to and stored in the frame memory 306, and is used as a predicted reference image of an image signal to be decoded next.

However, in the configuration of FIG. 23, the output of the arithmetic circuit 305 is output to the outside as described above, and is stored in the frame memory 306, and is used only as a prediction reference image of an image signal to be decoded thereafter. Instead, after being enlarged by the image enlargement circuit 327 to the same image size as the image signal of the upper layer, it is also used as a prediction reference image of the upper layer.

That is, the output of the arithmetic circuit 305 is output as a lower layer reproduced image as described above, and is also output to the frame memory 306. At the same time, the image enlargement circuit 327 outputs the same image size as that of the upper layer. , And output to the weighting circuit 328.

In the weighting circuit 328, the image enlargement circuit 3
27 is multiplied by (1−W) calculated using the decoded weight W, and this value is output to the arithmetic circuit 317.

On the other hand, the bit stream of the upper layer is
The data is supplied to the variable length decoding circuit 310 via the reception buffer 309, where the variable length code is decoded, and the quantization scale, the motion vector, the prediction mode, and the weight coefficient are decoded together with the DCT coefficient. The DCT coefficient decoded by the variable length decoding circuit 310 is inversely quantized by the inverse quantization circuit 311 using the same quantization scale, and then subjected to inverse DCT processing by the inverse DCT circuit 312, and further thereafter, the operation The signal is supplied to the circuit 313.

The motion compensation circuit 315 generates a predicted reference image according to the decoded motion vector and the prediction mode, and inputs the predicted reference image to the weighting circuit 316. The weighting circuit 316 multiplies the output of the motion compensation circuit 315 by the decoded weight W, and outputs the multiplication result to the arithmetic circuit 317.

The arithmetic circuit 317 adds the outputs of the weighting circuits 328 and 316, and outputs the obtained image signal to the arithmetic circuit 313. The arithmetic circuit 313 adds the output of the inverse DCT circuit 312 and the output of the arithmetic circuit 317 and outputs the result as a reproduced image of the upper layer.
14 to be used as a predicted reference image of an image signal to be decoded thereafter.

Although the above description is for the processing of the luminance signal, the processing of the chrominance signal is similarly performed. However, in this case, a motion vector obtained by halving the luminance vector for the luminance signal in the vertical and horizontal directions is used.

The MPEG system has been described above.
In addition, various high-efficiency coding schemes for moving images have been standardized. For example, a so-called ITU-T (Internationa
l Telecommunication Union-Telecommunication secto
r: Telecommunications Standardization Sector of the International Telecommunications Union), mainly uses H.264 as an encoding method for communications. 261 and H.E. 263 is defined. This H. 261 and H.E. 263
Is basically a combination of motion-compensated predictive coding and DCT transform coding as in the MPEG system. Although details such as header information are different, the coding apparatus and the decoding apparatus are the same.

[0058]

In MPEG2, spatial scalability has already been standardized, but its coding efficiency is hardly sufficient. Therefore, M
In the PEG4 system and other new coding systems, it is an issue to improve the coding efficiency of spatial scalability.

The spatial scalability in the MPEG2 system will now be described in some detail. In the scalable coding method, the lower layer is coded in the same manner as a normal coding method, that is, MP @ ML in the case of MPEG2. The upper layer uses an image of the lower layer at the same time and an image decoded immediately before the same layer as a reference image. At this time, the prediction modes of the lower layer and the upper layer are determined completely independently. Therefore,
Despite the information being transmitted in the lower layer, there is a case where the information is not used at all in the upper layer, but is predicted from the decoded image in the upper layer and encoded.
This is equivalent to transmitting information that can be shared between the upper layer and the lower layer completely independently.

Therefore, it is an object to minimize the duplication of information transmission as described above and to improve the coding efficiency.

In the MPEG2 system, the coding mode can be specified only on a macroblock basis. This is fine when dealing with images in relatively uniform areas,
When one macroblock contains a sequence that moves in a complicated manner or an image having a different property (for example, a still area and a moving area), it causes a reduction in coding efficiency.

Accordingly, the present invention has been made in view of such circumstances, and in a spatial scalable encoding method, it is possible to improve a prediction efficiency and to improve a coding efficiency by improving a moving picture signal. It is an object of the present invention to provide an encoding method and apparatus, and a moving image signal decoding method and apparatus.

[0063]

SUMMARY OF THE INVENTION The present invention has been proposed in order to solve the above-mentioned problem. An image signal is separated and hierarchized into at least two layers of high-resolution and low-resolution image signals. The method encodes a resolution and a low-resolution image signal using a predicted reference image, and obtains a change amount of a low-resolution image signal using motion information obtained from the high-resolution image signal. Compared with a predetermined threshold, when the difference is equal to or greater than the threshold, a predicted reference image is generated by adding the amount of change to a decoded image signal of a signal obtained by encoding a high-resolution image signal, and the generated predicted reference image is used. It is characterized by performing encoding.

That is, the present invention generates a new reference image by adding the change amount of the lower layer to the reference image of the upper layer, in addition to the prediction mode of the conventional MPEG system, and generates the new reference image. To realize inter-frame predictive coding. With this method, it is possible to improve the coding efficiency of spatial scalable coding and improve the image quality.

[0065]

Embodiments of the present invention will be described below with reference to the drawings.

First, before giving a detailed description of the embodiments of the present invention, the basic concept of the present invention will be described in connection with the properties of each reference image in MPEG.

Normally, the image signal of the upper layer has better image quality than the image signal of the lower layer. Therefore, when the change between frames is small, it is often the case that the motion compensation is performed using the image of the upper layer to reduce the prediction error. However, when the change between frames is large, the efficiency of motion compensation is reduced. Therefore, it is more efficient to use an image at the same time as a reference image. For this reason, an image signal from a lower layer at the same time may be used as a reference image in an area where the inter-frame difference is large, and an image of the same layer (upper layer) if the inter-frame difference is small. .

Next, the duplication of information of the spatial scalable coding of the MPEG system will be described.

Consider a case where a macroblock having an upper layer is encoded. At this time, when information (DCT coefficient) is transmitted in a macroblock of a lower layer corresponding to the macroblock of the upper layer, if prediction is performed with reference to the same image of the upper layer, the lower layer and the upper layer will be different. Information will be transmitted independently. That is, the information transmitted in the lower layer is not used for encoding in the upper layer, which causes a reduction in efficiency. Therefore,
When the DCT coefficients are transmitted in the corresponding lower layer macroblock, the use of the lower layer image as the prediction reference image reduces the duplication of data transmission and is more efficient.

In the scalable coding of the MPEG system, there are only three modes of prediction of the upper layer, namely, a prediction from an image of an upper layer, a prediction from a lower layer, and an average value thereof. Therefore, for example, when the image of the upper layer is used as the reference image, the information transmitted in the lower layer is not used at all in the upper layer.

As described above, the present invention improves the coding efficiency by introducing a new prediction method. Specifically, in the embodiment of the present invention, a new reference image is generated by adding the change amount of the lower layer to the reference image of the upper layer, in addition to the prediction mode of the conventional MPEG system, Inter-frame prediction coding is realized. With this method, it is possible to improve the coding efficiency of spatial scalable coding and improve the image quality. As an effect of the improvement, there is an effect of improving the signal-to-noise ratio (S / N) by about 0.5 dB as compared with the conventional one.

FIG. 1 shows an example of an image signal encoding apparatus according to an embodiment of the present invention.

In FIG. 1, an input image signal is first input to an image signal hierarchical circuit 1. This image signal layering circuit 1 separates an input image signal into a plurality of layers. In addition,
FIG. 1 shows the configuration in the case of two layers (one lower layer and one upper layer), but it is also possible to similarly separate into a plurality of layers. For the sake of simplicity, here, the case of separation into two layers will be described hereinafter.

For example, in the case of spatial scalability, the image signal layering circuit 1 converts the resolution of an input image signal and a key signal to generate image signals of a lower layer and an upper layer.

Further, for example, in the case of temporal scalability (scalability in the time axis direction), the image signal layering circuit 1 switches the output of the image signal to the lower layer and the upper layer according to the time and outputs. For example, in the case of FIG. 2, sequentially supplied images (pictures) VOP0
VOP6 is composed of images VOP0, VOP2, VOP4, VO
P6 is the lower layer and the images VOP1, VOP3, V
OP5 is output to the upper layer. In the case of the temporal scalability, resolution conversion such as enlargement or reduction of an image signal is not performed unlike the spatial scalability described above.

Further, for example, a so-called SNR (Signal to
In the case of Noise Retio) scalability, the image signal layering circuit 1 outputs the input image signal and the key signal as they are to each layer. That is, the same image signal and key signal are output to the lower layer and the upper layer.

In the present embodiment, the case of the above spatial scalability is taken as an example.

For example, in the case of spatial scalability, the image signal layering circuit 1 outputs an image signal obtained by reducing (resolution converting) the input image signal and the key signal as a lower layer, while outputting the input image signal to an upper layer. Output as is. The resolution conversion here is, for example, a reduction filtering process using a thinning filter. The image signal layering circuit 1 may output an image signal obtained by enlarging (resolution converting) the input image signal as an upper layer, and may output the input image signal to a lower layer as it is. The resolution conversion in this case is an enlargement filtering process using an enlargement filter or the like. Further, in the image signal layering circuit 1, two independently generated image signals (which may have different resolutions or have the same resolution) may be used.
Output may be made to the upper layer and the lower layer, respectively. In this case, which image is output to the upper layer and the lower layer is determined in advance. The image signal of the upper layer is sent to the upper layer encoding circuit 3 via the delay circuit 2, and the image signal of the lower layer is sent to the lower layer encoding circuit 5.

As described above, the image signal layering circuit 1 for converting the resolution of the input image signal and outputting the converted image signal to the lower layer and the upper layer also has a flag FR indicating the magnification of the resolution of the upper layer image with respect to the lower layer image. Is also output.
The flag FR is sent to the resolution conversion circuit 4 and the upper layer encoding circuit 3 via the delay circuit 2. The resolution conversion circuit 4 is provided separately from the resolution conversion means provided in the image signal hierarchical circuit 1.

Here, a specific configuration of the lower layer coding circuit 5 will be described with reference to FIG.

In FIG. 3, the input image signal of the lower layer supplied to the lower layer coding circuit 5 is first input to the frame memory 21, read out in a predetermined order, and coded by the configuration of the subsequent stage. . The image data to be encoded is stored in a frame memory 21 in macroblock units.
And input to the motion vector detection circuit 22. The motion vector detection circuit 22 processes the image data of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance. Whether the image of each frame input sequentially is processed as one of I, P, and B pictures is predetermined (for example, I, B, P, B,
P,..., B, P).

Here, the motion vector detection circuit 22 refers to a predetermined reference frame (a forward original image, a backward original image, an original frame image) and performs motion compensation.
The motion vector is detected. The motion compensation (inter-frame prediction) has three types of prediction modes: forward prediction, backward prediction, and bidirectional prediction. The prediction mode of a P picture is only forward prediction, and the prediction mode of a B picture is of three types: forward prediction, backward prediction, and bidirectional prediction. The motion vector detection circuit 22 selects a prediction mode that minimizes a prediction error, and generates a prediction vector at that time.

At this time, the prediction error is compared with, for example, the variance value of the macroblock to be coded. If the variance value of the macroblock is smaller, the prediction is not performed on that macroblock, and the intra-frame coding is not performed. Done. In this case, the prediction mode is intra-picture encoding (intra). The motion vector and the information on the prediction mode are input to the variable length coding circuit 26 and the motion compensation circuit 32.

The motion compensation circuit 32 generates a predicted reference image based on a predetermined motion vector, and supplies the predicted reference image signal to the arithmetic circuit 23. The arithmetic circuit 23 obtains a difference between the value of the macroblock to be encoded from the frame memory 21 and the value of the predicted reference image from the motion compensation circuit 32, and outputs the difference signal to the DCT circuit 24. In the case of an intra macro block, the arithmetic circuit 2
3 outputs the macroblock signal to be encoded to the DCT circuit 24 as it is.

The DCT circuit 24 converts the difference output into a DC
T (Discrete Cosine Transform) processing is performed to convert to DCT coefficients. The DCT coefficient is input to the quantization circuit 25 and is quantized in a quantization step corresponding to the amount of data stored in the transmission buffer 27 (the remaining amount of data that can be stored in the buffer). Is entered.

The variable-length encoding circuit 26 includes a quantization circuit 25
The image data (in this case, I-picture data) supplied from the quantization circuit 25 is converted into a variable-length code such as a Huffman code in accordance with the supplied quantization step (quantization scale). Then, the data is output to the transmission buffer 27.

The variable length coding circuit 26 also includes a quantization step (quantization scale) from the quantization circuit 25 and a prediction mode (intra-picture prediction, forward prediction, backward prediction) from the motion vector detection circuit 22. , Or a mode indicating which of the bidirectional predictions has been set) and a motion vector, and these are also variable-length coded.

The transmission buffer 27 temporarily stores the input data, and feeds back data corresponding to the stored amount to the quantization circuit 25 as a quantization control signal. That is, when the data storage amount (remaining amount of data that can be stored) increases to the allowable upper limit, the transmission buffer 27 increases the quantization scale of the quantization circuit 25 by the above-described quantization control signal. 25, the data amount of the quantized data output from the control unit 25 is reduced. Conversely, when the data storage amount (remaining amount of data that can be stored) decreases to the allowable lower limit, the transmission buffer 27 reduces the quantization scale of the quantization circuit 25 by the quantization control signal. , The amount of quantized data output from the quantization circuit 25 is increased. In this way, overflow or underflow of the transmission buffer 27 is prevented.

The data stored in the transmission buffer 27 is read out at a predetermined timing and output to the transmission path as a bit stream.

On the other hand, the data output from the quantization circuit 25 is also input to the inverse quantization circuit 28. In the inverse quantization circuit 28, the data supplied from the quantization circuit 25 is inversely quantized according to the quantization step also supplied from the quantization circuit 25. The output of the inverse quantization circuit 28 is input to an IDCT (inverse DCT) circuit 29, where it is subjected to inverse DCT processing, and then sent to an arithmetic circuit 30.
In this arithmetic circuit 30, the output of the motion compensation circuit 32 and the ID
The image signal is restored by adding the output of the CT circuit 29.
This image signal is supplied to the frame memory 31 and stored. The motion compensation circuit 32 generates a prediction reference image using the image in the frame memory 31, the motion vector, and the prediction mode.

The output of the operation circuit 30, that is, the decoded image signal is also supplied to the upper layer encoding circuit 3 of FIG.

The frame memory 31 reads out a predetermined locally decoded image and outputs it to the resolution conversion circuit 4 in FIG. 1 in accordance with the encoding in the upper layer encoding circuit 3 in FIG.

Returning to FIG. 1, the resolution conversion circuit 4 performs the above-described lower layer encoding circuit 5 in accordance with the flag FR indicating the magnification of the resolution of the upper layer image with respect to the lower layer image.
Is converted by the filtering process (enlargement process in this case) as described above, and is supplied to the upper layer encoding circuit 3. When the magnification is 1, that is, when the size of the upper layer is equal to the size of the lower layer, the resolution conversion circuit 4 outputs the image signal without any operation.

The image signal of the upper layer generated by the image signal layering circuit 1 is supplied to the upper layer encoding circuit 3 via the delay circuit 2. The delay circuit 2 delays the image signal of the upper layer by the time required for encoding the image of the predetermined lower layer in the lower layer encoding circuit 5.

Next, a specific configuration of the upper layer coding circuit 3 will be described with reference to FIG.

In FIG. 4, the input image signal of the upper layer supplied to the upper layer coding 3 is first input to and stored in the frame memory 41, read out in a predetermined order, and encoded by the subsequent stage. Be transformed into The image data to be encoded is read from the frame memory 41 in macroblock units, and input to the motion vector detection circuit 42. The motion vector detection circuit 42 processes the image data of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance. It is determined in advance whether an image of each sequentially input frame is processed as a picture of I, P, or B (for example, I, P, or B).
B, P, B, P,... B, P are processed in this order).

Here, in the case of the spatial scalability, the image signals of the upper layer and the lower layer are coded as shown in FIG. 5, for example. 5, first image VOP21 _M of the upper layer is encoded as a P picture. The reference image at this time is an image V of the lower layer at the same time.
OP21 _L is obtained. The image VO of the second or higher layer
P22 _M ~VOP24 _M is encoded as B pictures. In this case, these images VOP22 _M ~VOP24 _M
Are the images of the immediately preceding upper layer and the lower layer images VOP22 _L , VOP23 _L ,
It encoded using VOP24 _L as the reference image. Also, in this case, the B picture of the upper layer becomes a reference image when encoding another image VOP, like the P picture of the lower layer. Note that SNR scalability is a special case of spatial scalability, in which the size of the upper layer is equal to that of the lower layer, and the coding procedure is the same.

In the case of temporal scalability, encoding is performed, for example, as shown in FIG.
In FIG. 6, an upper layer image VOP1 is encoded as a B picture, and the lower layer image VOP0 and image VOP2 are used as reference images for this image VOP1. In the case of the image VOP3 of the upper layer, it is encoded as a B picture, and the image VOP1 of the immediately preceding upper layer and the image VOP4 of the lower layer are used as reference images. Also,
In the case of the upper layer image VOP5, it is encoded as a P picture, and the immediately preceding upper layer image VOP3 is used as a reference image.

Hereinafter, reference pictures of P and B pictures in the upper layer will be described.

In the prediction of the upper layer, not only the image of the same layer but also the image of another layer (scalable layer) can be used as the reference image. For example, in the case of scalability of two layers as in the present embodiment, the upper layer can use the image of the lower layer as a reference image.

The P picture is subjected to forward prediction, and an image of the same layer or an image of another layer can be used as a reference image. In a B picture, an image of the same layer or an image of another layer can be used as a reference image for forward prediction and backward prediction.

Here, the prediction mode in the coding of the upper layer will be described.

The prediction mode is determined for each macroblock, as in the coding of the lower layer. Upper layer P
There are the following two types of prediction modes for macroblocks of pictures.

(1) Intra-frame (intra) coded macroblock (2) Forward prediction (forward) macroblock The above prediction mode is the same as the prediction mode of the macroblock in the lower layer.

There are the following types of macroblock prediction modes for B pictures in the upper layer.

(1) Intra-frame prediction (intra) macroblock (2) Forward prediction (forward) macroblock (3) Backward prediction (backward) macroblock (4) Bidirectional prediction (bidirectional) macroblock 5) Two-layer prediction macroblock Among the above prediction modes, the prediction modes shown in (1) to (4) are the same as the prediction mode of the macroblock of the lower layer.

The two-layer prediction macroblock shown in (5) will be described with reference to FIGS. 4 and 7.

In FIG. 7, images VOP11 and VOP12 indicate lower layer images, and images VOP13 and VOP14 indicate upper layer images. Images VOP11 and VOP13 are images at the same time, and images VOP12 and VOP14
Are images at the same time. The image VOP11 is an I or P picture, and the image VOP12 is a P picture. The image VOP13 is a P or B picture, and the image VOP1 is
4 is a B picture. The images vm5 and vm6 are images obtained by converting the resolution of the lower layer image by the resolution conversion circuit 4 in FIG.

The motion vector detecting circuit 4 shown in FIG.
2 detects a motion vector using the image VOP13 of the upper layer as a reference image. This motion vector is shown in FIG.
The motion compensation prediction images PM11 and PM7 of the upper layer and the lower layer of FIG. 7 are formed. That is, the motion compensation circuit 52 reads a predetermined image signal of the image VOP13 of the upper layer from the frame memory 51 as the motion compensation predicted image PM11 using the motion vector. In the motion compensation circuit 53, a predetermined image signal of the enlarged image vm5 enlarged by the resolution conversion circuit 4 in FIG. 1 and stored in the frame memory 54 is similarly subjected to the motion compensation prediction using the motion vector. Read as image PM7. Further, the frame memory 54 also stores an enlarged image vm6 enlarged by the resolution conversion circuit 4 of FIG.

The motion-compensated predicted image PM7 read after being motion-compensated by the motion-compensating circuit 53 is sent to the arithmetic circuit 57 of FIG. The difference between them is calculated. The difference signal D obtained by the difference operation in the operation circuit 57
V8 is compared with a predetermined threshold value TH for each pixel by a threshold value comparison circuit 58. In the example of FIG. 4, the operation circuit 57 is provided independently after the motion compensation circuit 53, but the function of the operation circuit 57 may be provided to the motion compensation circuit 53. It is.

When each pixel of the difference signal DV8 is smaller than the threshold value TH, the threshold value comparison circuit 58 takes out the value of the pixel of the difference signal DV8 as 0. Conversely, if the difference is equal to or greater than the threshold value TH, the value of the pixel of the difference signal DV8 is extracted as it is. In the present embodiment, the threshold TH is set to a certain value in advance, but may be set to an arbitrary value. When the threshold value TH is 0, each pixel value of the difference signal DV8 is extracted as it is.
Further, the threshold value can be changed depending on the color. In the example of FIG. 4, the threshold comparison circuit 58 is provided as an independent configuration. However, the function of the threshold comparison circuit 58 may be provided to, for example, the motion compensation circuit 53.

The motion-compensated predicted image PM11 obtained by performing motion compensation in the motion compensation circuit 52 is sent to the arithmetic circuit 59. The arithmetic circuit 59 adds the motion compensated prediction image PM11 and the output image from the threshold value comparison circuit 58 for each pixel to generate an image AV10.
In the two-layer prediction macro block, the arithmetic circuit 59
Is used as the prediction reference image. That is, when the difference is less than the threshold value TH, the motion-compensated predicted image PM11 itself is a prediction reference image. Note that, in the example of FIG.
Is provided independently at the subsequent stage of the motion compensation circuit 52, but the function of the arithmetic circuit 59 can be provided to the motion compensation circuit 52.

FIG. 8 shows a flow of generating a prediction reference image of a two-layer prediction macroblock.

In FIG. 8, in step ST11, the enlarged image vm5 and the enlarged image vm6 of the decoded image of the lower layer are stored in the frame memory. Note that the frame memory 54 can be provided separately for the enlarged image vm5 and for the vm6.

In step ST12, the motion vector detecting circuit 42 calculates a motion vector using the reference image of the upper layer.

In step ST13, the motion compensation circuit 53 performs motion compensation on the enlarged image vm5 of the lower layer using the motion vector of the upper layer, generates the motion compensated prediction image PM7, and An inter-frame difference is calculated between the enlarged image vm6 of the lower layer and the motion compensated prediction image PM7, and the obtained difference image DV8 is output.

In step ST14, the difference image DV
The threshold value TH is compared with the absolute value of the pixel 8 and the threshold value TH.
If it is less than the above, the process proceeds to step ST15, and if it is above, the process proceeds to step ST16.

When the absolute value of each pixel value of the difference image DV8 is less than the threshold value TH, each pixel value of the difference image DV8 is set to 0. The motion-compensated signal, that is, the motion-compensated predicted image PM11 is output as a predicted reference image.

On the other hand, when the absolute value of each pixel value of the difference image DV8 is equal to or larger than the threshold value TH, each pixel value of the difference image DV8 is sent to the arithmetic circuit 59 as it is. Motion compensated prediction image P
A signal obtained by adding the difference image DV8 of the lower layer to M11 is output as a prediction reference image.

Here, the image signal of the motion-compensated predicted image PM11, that is, the image signal of the motion-compensated image VOP13 is represented by a _i
And, an image signal of the macroblock in the corresponding position of the enlarged image vm6 and b _i, the motion compensated prediction picture PM
7 That enlarged image vm5 which is motion compensated and c _i,
Assuming that the difference image DV8 is diff, the prediction reference image of the two-layer prediction macroblock is given by the following equation (1).

[0121] _{diff = (b i- c i)} a i + (b i -c i), (| diff | ≧ TH) a i, (| diff | <TH) (1) as described above, two layers The reason why the efficiency is improved by introducing the prediction macroblock will be described.

As described above, the lower layer image has a lower resolution than the upper layer image. Therefore, the enlarged images vm5 and vm6 each have information on low-frequency components with respect to the image of the upper layer. Therefore, the difference between the enlarged images vm5 and vm6 (the difference image D
V8) are images VOP13 to VOP14 of the upper layer.
It contains information on the amount of change in the low-frequency component in the time change to. Therefore, using the difference between these enlarged images vm5 and vm6 means that the predicted reference image is generated including the variation of the low frequency component as compared with the case where the image VOP13 is simply predicted. The efficiency of prediction is improved. If the difference between the enlarged images vm5 and vm6 (the difference image DV8) is larger than a certain value (threshold TH), it means that information has been transmitted in the lower layer. By utilizing this amount of change for prediction in the upper layer, it is possible to reduce duplication of information in the upper layer and the lower layer, thereby improving coding efficiency.

In the above-mentioned two-layer prediction macroblock, a motion vector is subjected to variable-length coding and transmitted like the forward prediction macroblock.

Next, in the above-mentioned five macroblock prediction modes for the B picture of the upper layer, the mode that minimizes the prediction error is selected. The flag indicating the prediction mode is encoded and transmitted by the variable length encoding circuit.

As described above, in the prediction in the upper layer, an image of a scalable layer different from the image to be coded, for example, an image of a lower layer having a lower resolution can be used as a reference image for generating a predicted reference image. . Therefore, the upper layer needs to transmit a flag indicating which layer image is used as a reference image for generating a predicted reference image. Therefore, here, a flag (a syntax identifier (ref_layer_i to be described later) indicating which layer image other than the same layer has been used for generating a predicted reference image for each scalable layer is used.
d)) is set, encoded, and transmitted. In addition, for each image (the VOP), a flag indicating from which layer the forward (forward) prediction and the backward (backward) prediction are predicted based on the flag (ref_layer_id) (a syntax identifier (described later)). ref_select_cod
e) Set, encode and transmit. The flag (ref_select_code) in the P picture is shown in the table of FIG. Also,
FIG. 1 shows a flag (ref_select_code) in a B picture.
0 is shown in the table. Details of the syntax will be described later.

Reference images for the upper layer and the lower layer may be set freely within the range permitted by the tables shown in FIGS. 9 and 10 in addition to FIGS. 5 and 2. FIG.
10 and the syntax of the table of FIG. 10, there is no explicit distinction between spatial scalability and temporal scalability.

Here, in the case of a P picture, the flag (re
When f_select_code) is “11”, the flag (ref_layer
_id) is used as a reference image for generating a predicted reference image at the same time of the layer (VOP). This is used for spatial scalability and SNR scalability. Other modes are used for temporal scalability.

In the case of a B picture, a flag (ref_select
If (_code) is “00”, the image (VOP) decoded immediately before the same layer as the same-time image (VOP) of the layer indicated by the flag (ref_layer_id) is used as a reference image for generating a predicted reference image. This is used for spatial scalability and SNR scalability. Other modes are used for temporal scalability.

Each image (VOP) of each layer is composed of I, P,
Which type of B picture is to be encoded is determined in advance. The motion vector detection circuit 42 shown in FIG. 4 uses a flag (ref_layer_i) based on a preset picture type.
d), (ref_select_code) is set and the motion compensation circuit 52
And outputs it to the variable length coding circuit 46.

The decoded image signal of the image (VOP) of the lower layer is supplied to the upper layer encoding circuit 3 via the resolution conversion circuit 4 of FIG. 1, and is supplied to the frame memory 54 of FIG.

The motion vector detection circuit 42 sets a predetermined reference frame to a flag (ref_layer_id).
Reference is made from the frame memory 41 or the frame memory 51 based on the and the flag (ref_select_code) to perform motion compensation and detect the motion vector. There are three types of motion compensation (inter-frame prediction): forward prediction, backward prediction, and bidirectional prediction. The prediction mode of the P picture is only forward prediction, and the prediction modes of the B picture are four types of forward prediction, backward prediction, bidirectional prediction, and the two-layer prediction. Motion vector detection circuit 42
Selects a prediction mode that minimizes the prediction error and generates a prediction vector at that time.

In the two-layer prediction mode, similarly to the forward prediction, a motion vector is detected by referring to the image signal recorded in the frame memory 51.

At this time, the prediction error is compared with, for example, the variance of the macroblock to be coded. If the variance of the macroblock is smaller, the prediction is not performed on the macroblock and the intra-frame coding is performed. In this case, the prediction mode is intra-picture encoding (intra). The motion vector and the prediction mode are input to the variable length coding circuit 46 and the motion compensation circuit 53.

The motion vector detecting circuit 42 is also supplied with a flag FR indicating how many times the size (resolution) of the upper layer is lower than the lower layer. According to the table shown in FIG. 10, in the case of a B picture (image VOP), the spatio-temporal scalability of the flag (ref_select_code == "00") is used. , Forward (forward) prediction is an image (VOP) decoded using backward prediction immediately before the same layer
From the prediction. If the flag FR indicating the magnification is 1 (the resolution of the lower layer is equal to the resolution of the upper layer) and the flag (ref_select_code == "00"), this is a special case of spatial scalability, and SNR scalability Is shown. In this case, for the forward prediction of the upper layer, the motion vector and the prediction mode used by the VOP of the lower layer at the same time are used as they are. Therefore, in this case, the motion vector detection circuit 42 supplies the motion vector and the prediction mode supplied from the lower layer to the motion compensation circuit 52. In this case, the variable length coding circuit 46 does not code the motion vector.

In a B picture (VOP), the two-layer prediction mode is used only when the flag (ref_select_code == "00") is set. That is, in the B picture, the two-layer prediction mode is used only when the image of the lower layer at the same time as the upper layer is used as the reference image.

In the motion compensating circuit 52, the frame memory 51 and the motion compensating circuit 5 are operated based on a predetermined motion vector.
Then, a prediction reference image is generated from the output of No. 3 and input to the arithmetic circuit 43.

The arithmetic circuit 43 converts the difference signal between the value of the macroblock to be coded and the value of the prediction reference image into the DCT circuit 24
Output to In the case of an intra macro block, the arithmetic circuit 43 directly converts the macro block signal to be encoded into a DCT signal.
Output to the circuit 44.

In the DCT circuit 44, DCT (Discrete Cosine Transform) processing is performed, and the data is converted into DCT coefficients. This DC
The T coefficient is input to the quantization circuit 45, and the transmission buffer 4
After being quantized in a quantization step corresponding to the data storage amount (buffer storage amount) of No. 7, the data is input to the variable length encoding circuit 46.

The variable length coding circuit 46 includes a quantization circuit 45
The image data (in this case, I-picture data) supplied from the quantization circuit 45 is converted into a variable-length code such as a Huffman code in accordance with the supplied quantization step (scale) and transmitted. Output to the buffer 47.

The variable length coding circuit 46 also receives a quantization step (scale) from the quantization circuit 45 and a prediction mode (intra-picture prediction, forward prediction, backward prediction, bidirectional prediction or 2 prediction) from the motion vector detection circuit 42. A mode indicating which of the hierarchical predictions is set) and a motion vector are input, and these are also subjected to variable-length coding.

The variable length coding circuit 46 also includes a flag (identifier (FSZ_E) described later) indicating the size of the image (VOP) of the upper layer and a flag (identifier (FPOS_E) described later) indicating the position in absolute coordinates. Input, and these are also encoded.

The variable length encoding circuit 46 also receives a flag FR indicating how many times the resolution of the upper layer is higher than the resolution of the lower layer, and this is also encoded.

The transmission buffer 47 temporarily stores the input data, and outputs data corresponding to the stored amount to the quantization circuit 45.

The transmission buffer 47 increases the quantization scale of the quantization circuit 45 by the quantization control signal when the storage amount (remaining amount of storable data) increases to the permissible upper limit value. Decrease the amount of data. Conversely, when the storage amount (the remaining amount of data that can be stored) decreases to the allowable lower limit, the transmission buffer 47
Increases the data amount of the quantized data by reducing the quantization scale of the quantization circuit 45 by the quantization control signal. In this way, overflow or underflow of the transmission buffer 47 is prevented.

The data stored in the transmission buffer 47 is read at a predetermined timing and output to a transmission path.

On the other hand, the data output from the quantization circuit 45 is also input to the inverse quantization circuit 48, where it is inversely quantized in accordance with the quantization step supplied from the quantization circuit 45. The output of the inverse quantization circuit 48 is IDCT (inverse D
CT) circuit 49, where it is subjected to inverse DCT processing and then stored in the frame memory 51.

Returning to FIG. 1, the output bit streams of the upper layer coding circuit 3 and the lower layer coding circuit 5 are
The signal is input to the multiplexing circuit 6. The multiplexing circuit 6 multiplexes the bit streams of the lower layer and the upper layer and outputs the multiplexed bit stream.

Next, FIG. 11 shows an example of an image signal decoding apparatus according to the embodiment corresponding to the image signal encoding apparatus shown in FIG.

In FIG. 11, the bit stream is first input to a demultiplexing circuit 81. Demultiplexing circuit 81
Then, the bit stream is demultiplexed, that is, the bit stream is separated into an upper layer and a lower layer bit stream and output.

The bit stream of the lower layer is supplied to the lower layer decoding circuit 85 as it is. The bit stream of the upper layer is supplied to the upper layer decoding circuit 83 via the delay circuit 82.

In the delay circuit 82, the lower layer decoding circuit 85
Then, after delaying the time required to decode one image (for one VOP), the upper layer decoding circuit 83 is supplied with the upper layer bit stream.

The specific configuration of the lower layer decoding circuit 85 will be described with reference to FIG.

After the lower layer bit stream is temporarily stored in the reception buffer 91, it is supplied to the variable length decoding circuit 92. The variable length decoding circuit 92 performs variable length decoding on the data supplied from the reception buffer 91, outputs the motion vector and the prediction mode to the motion compensation circuit 97, and outputs the quantization step to the inverse quantization circuit 93, respectively. , And outputs the decoded image data to the inverse quantization circuit 93.

The variable length decoding circuit 92 also outputs the image (VO
Flag indicating the size of P) (identifier (FSZ_B) described later)
Is decoded and output to the motion compensation circuit 97 and the frame memory 96. The flag (FSZ_B) is also supplied to the upper layer decoding circuit 83 in FIG.

The inverse quantization circuit 93 includes a variable length decoding circuit 92
The image data supplied from the variable length decoding circuit 9
Inverse quantization according to the quantization step supplied from 2,
Output to the IDCT circuit 94. The data (DCT coefficient) output from the inverse quantization circuit 93 is subjected to inverse DCT processing in the IDCT circuit 94 and then supplied to the arithmetic circuit 95.

When the image data supplied from the IDCT circuit 94 is I-picture data, the data is output from the arithmetic circuit 95, and the image data (P or B picture Frame memory 96 for generating predicted reference image data
And stored. This data is output to the outside as a reproduced image as it is.

When the input bit stream is a P or B picture, the motion compensation circuit 97 generates a prediction reference image according to the motion vector and the prediction mode supplied from the variable length decoding circuit 92, and outputs it to the arithmetic circuit 95. The arithmetic circuit 95 adds the image data input from the IDCT circuit 94 and the predicted reference image data supplied from the motion compensation circuit 97 to obtain an output image. In the case of a P picture, the output of the arithmetic circuit 95 is also input to and stored in the frame memory 96, and is used as a reference image of an image signal to be decoded next.

The output of the arithmetic circuit 95 is also supplied to the upper layer decoding circuit 83 in FIG.

A flag (FSZ_) indicating the size of the image (VOP) of the lower layer decoded by the variable length decoding circuit 92
B) is supplied to the upper layer decoding circuit 83 in FIG.

Returning to FIG. 11, the upper layer bit stream demultiplexed by the demultiplexing circuit 81 is supplied to the upper layer decoding circuit 83 via the delay circuit.

The specific configuration of the upper layer decoding circuit 83 will be described with reference to FIG.

After the bit stream of the upper layer is temporarily stored in the reception buffer 101, the bit stream of the variable length decoding circuit 10
2 is supplied. The variable length decoding circuit 102 performs variable length decoding on the data supplied from the reception buffer 101, outputs the motion vector and the prediction mode to the motion compensation circuit 107, and outputs the quantization step to the inverse quantization circuit 103, respectively. An inverse quantization circuit 103 decodes the decoded image data.
Output to

The motion vector and the prediction mode are also output to the motion compensation circuit 119.

The variable length decoding circuit 102 also outputs the image (VO
The flag (FSZ_E) indicating the size of P) is decoded and output to the motion compensation circuit 107 and the frame memory 106.

The variable length decoding circuit 102 also provides an upper layer image (VOP) with respect to a lower layer image (VOP).
Decoding the flag FR indicating the magnification of the size (resolution) of
The signal is output to the motion compensation circuit 107 and the resolution conversion circuit 84 in FIG.

The resolution conversion circuit 84 in FIG. 11 converts the resolution of the decoded image signal of the lower layer image (VOP) and the key signal by filtering according to the flag FR indicating the magnification, and the upper layer decoding circuit 8
3 is supplied to the frame memory 118.

The variable-length decoding circuit 102 also sets a flag (identifier (ref_lay described later) indicating a layer used for reference of prediction.
er_id) and (ref_select_code)) are decoded and output to the motion compensation circuit 107.

The inverse quantization circuit 103 includes the variable length decoding circuit 1
The image data supplied from 02 is inversely quantized according to the quantization step also supplied from the variable length decoding circuit 102, and is output to the IDCT circuit 104. Inverse quantization circuit 1
03 (DCT coefficient) is the IDCT
The inverse DCT processing is performed by the circuit 104, and the result is supplied to the arithmetic circuit 105.

When the image data supplied from the IDCT circuit 104 is I-picture data, the data is output from the arithmetic circuit 105 and is input to the arithmetic circuit 105 later (P or B picture data). To generate the predicted reference image data of the frame memory 10
6 and stored. This data is output to the outside as a reproduced image as it is.

When the input bit stream is a P or B picture, the motion compensation circuit 107
The flags (ref_layer_id), (ref_layer_id) and (ref_se
lect_code), a prediction reference image is generated from the frame memory 106 and the frame memory 118, and the arithmetic circuit 1
Output to 05. In the arithmetic circuit 105, the IDCT circuit 10
4 is added to the prediction reference image data supplied from the motion compensation circuit 107 to obtain an output image. In the case of a P picture, the output of the arithmetic circuit 105 is also input to and stored in the frame memory 106, and is used as a predicted reference image of an image signal to be decoded next.

In the decoding circuit 83 of the upper layer in FIG.
When the prediction mode is a macroblock in the two-layer prediction mode, a prediction reference image is generated in the encoding circuit in the same manner as the method of generating the output of the arithmetic circuit 59 in FIG.

More specifically, the output signal of the arithmetic circuit 95 in the lower layer image signal decoding circuit of FIG. 12 is expanded by the resolution conversion circuit 84 of FIG. Input to the decoding circuit 83
Is supplied to the frame memory 118 in FIG.

The motion compensation circuit 119 in FIG. 13 refers to the frame memory 118 based on the decoded motion vector, outputs the difference between the enlarged images vm5 and vm6 in FIG. 7 (difference image DV8), and outputs the difference to the threshold value comparison circuit 120. send. In the example of FIG. 13, the motion compensation circuit 119 obtains the difference, but an arithmetic circuit for calculating the difference may be provided outside.

At this time, in the threshold value comparing circuit 120,
When the absolute value of the difference from the motion compensation circuit 119 is smaller than a predetermined value (that is, the threshold value TH), the pixel value of the difference is set to 0. Conversely, if the difference is equal to or greater than the threshold value TH, the difference value is supplied to the motion compensation circuit 107 as it is. It should be noted that the threshold value comparison circuit 120 includes a motion compensation circuit 119.
Alternatively, it can be included in the motion compensation circuit 107.

In the motion compensation circuit 107, the prediction mode is 2
If it is a hierarchical prediction macroblock, the frame memory 1
The prediction reference image represented by the above equation (1) is generated from the output of the reference number 06 and the output of the motion compensation circuit 119 via the threshold value comparison circuit 120, and is output to the arithmetic circuit 105. In other prediction modes, the motion compensation circuit 107 operates similarly to the motion compensation circuit in the lower layer. Note that FIG.
In the example of (1), the output of the threshold value comparison circuit 120 is added in order to obtain a predicted reference image in the motion compensation circuit 107. However, an arithmetic circuit for performing the addition may be provided externally.

The motion compensation circuit 107 sets the flag FR indicating the magnification to 1 and sets the flag (ref_select_code ==
'00'), the image (VO) of the lower layer at the same time
A prediction image is generated using the motion vector and the prediction mode supplied from P), and output to the arithmetic circuit 105.

In the two-layer prediction macroblock, the same processing is performed for both the luminance and color difference signals.

Next, the syntax of the scalable coding will be described. The following MPEG4 VM (Verification Mode
l) will be described as an example.

FIG. 14 shows the configuration of a bit stream. VS (Video Session Class) is one or more VOs
(Video Object Class) is a set of bit streams.

The syntax of the VS is shown below. This syntax conforms to the so-called C ++.

Syntax No. of bits Mnemonic VideoSession () {video_session_start_code sc + 8 = 32 do * {Videoobject ()} while (nextbits_bytealigned () == video_object_start_code) next_start_code () video_session_end_code sc + 8 = 32} Next, VO ( The syntax of Video Object Class) is shown below.

Syntax No. of bits Mnemonic Videoobject () {video_object_start_code sc + 3 = 27 video_object_id 5 do {VideoObjectLayer ()} while (nextbits_bytealigned () == video_object_layer_start_code) next_start_code ()} VO is the whole image or a part of the image Is a bit stream of the object. VOL (Video Object Layer Class) is a class for scalability.

The syntax of the VOL is shown below.

Syntax No. of bits Mnemonic VideoObjectLayer () {video_object_layer_start_code sc + 4 = 28 video_object_layer_id 4 video_object_layer_shape 2 if (video_object_layer_shape == "00") {video_object_layer_width 10 video_object_layer_width = video_object_layer_shape_effects_layer () || (video_object_layer_shape_effects == "0011") || (video_object_layer_shape_effects == "0100") || (video_object_layer_shape_effects == "0101") video_object_layer_feather_dist 3 if ((video_Object_layer_shape_effects == "0100") || (video_object_layer = ") {for (i = 0; i <video_object_layer_feather_dist; i ++) feathering_filter (); 8 * 15} video_object_layer_sprite_usage 2 if (video_object_layer_sprite_usage! = SPRITE_NOT_USED) {if (video_object_layer_sprite_usage_ = sprite_usage_sprite_usage == ON-LINE_SPRIvrite_sprite_usage_ = sprite_usage 13 for (i = 0; i <no_of_sprite_points; i ++) {sprite_point [i] _x_coordinate 13 sprite_point [i] _y_coordinate 13} lighting_change_in_sprite 1} v ideo_object_layer_quant_type 1 if (video_object_layer_quant_type) {load_intra_quant_mat 1 if (load_intra_quant_mat intra_quant_mat [64] 8 * 64 load_nonintra_quant_mat 1 if (load_nonintra_quant_mat nonintra_quant_mat [64] 8 * 64} Error_resilient_disable 1 Intra_acdc_pred_disable 1 video_object_layer_fcode_forward 2 video_object_layer_fcode_backward 2 Separate_motion_shape_texture 1 if (video_object_layer_sprire_usage == STATIC_SPRITE) sprite_shape_texture () Scalability 1 if (scalability) {ref_layer_id 4 ref_layer_sampling_direc 1 hor_sampling_factor_n 5 hor_sampling_factor_m 5 vert_sampling_factor_n 5 vert_sampling_factor_m 5 enhancement_type 1} do {VideoObjectPlane () _ start_video (_) _ start_video () Identified by the number shown. For example, (video_object_lay
VOL0 where er_id = 0) is a lower layer, and VOL1 where (video_object_layer_id = 1) is an upper layer, for example. The number of scalable layers may be arbitrary.

The scalability is a 1-bit flag indicating whether the VOL is a lower layer or an upper layer. If (scalability = 1), the VOL is a lower layer, otherwise it is an upper layer.

The identifier (ref_layer_id) is a flag indicating the number of a VOL used as a reference image other than the own VOL. It is transmitted only to upper layers.

Identifiers (hor_sampling_factor_n), (hor
_sampling_factor_m) indicates how many times the horizontal length of the upper layer is greater than the horizontal length of the lower layer (indicating the magnification of the horizontal resolution). The horizontal dimension of the upper layer relative to the lower layer is given by the following equation.

Hor_sampling_factor_n / hor_sampling_factor_m identifier (ver_sampling_factor_n), (ver_sampling_fa
ctor_m) indicates how many times the vertical length of the upper layer is greater than the vertical length of the lower layer (indicating the magnification of the vertical resolution). The vertical dimension of the upper layer relative to the lower layer is given by the following equation.

Ver_sampling_factor_n / ver_sampling_factor_m The VOP (Video Object Plane Class) syntax is shown below.

Syntax No.of bits Mnemonic VideoObjectPlane () {VOP_start_code sc + 8 = 32 do {modulo_time_base 1} while (modulo_time_base! = "0") VOP_time_increment 10 VOP_prediction_type 2 if ((video_object_layer_sprite_usege! = SPRITE_NOT_USE_VID) SPRITE)) {if (no_of_sprite_points> 0) {encode VOP_points ()} if (lighting_change_in_sprite) {lighting_change_factor_encode ()} if (video_object_layer_sprite_useage == STATIC_SPRITE) {return ()} else if (video_object_layer_sprite_us = 8) }} if (video_object_layer_shape! = "00") {VOP_width 10 VOP_heigth 10 VOP_horizontal_mc_spatial_ref 10 marker_bit 1 VOP_vertical_mc_spatial_ref 10 if (scalability && enhancement_type) background_composition 1} disable_sadct 1 if (VOP_prediction) VOP_predictionquantity = type (video_object_layer_shape_effects == "0011") || (video_object_layer_shape_effects == "0101")) {VOP_const ant_alpha 1 if (VOP_constant_alpha) VOP_constant_alpha_value 8} if (! scalability) {if (! separate_motion_shape_texture) if (error_resilience_disable) combined_motion_shape_texture_coding () else do {do {combined_motion_shape_texture_coding ()} 0000 () ext. nextbits_bytealigned ()! = 000 0000 0000 0000 0000 0000) {next_resync_marker () resync_marker 17 macroblock_number 1-12 quant_scale 5}} while (nextbits_bytealigned ()! = 000 0000 0000 0000 0000 0000) else {if (video_object_layer_shape! = "00" ) {do {first_shape_code 1-3} while (count of macroblocks! = total number of macroblocks)} if (error_resilience_disable) {motion_coding () if (video_object_layer_shape! = "00") shape_coding () texture_coding ()} else do {do {motion_coding ()} while (next_bits ()! = "1010 0000 0000 0000 1") motion_marker 17 if (video_object_layer_shape! = "00") shape_coding () do {texture_coding ()} while (nextbits_bytealigned ()! = "0000 0000 0000 0000 ") if (nextbits_bytealigned ()! = "000 0000 0000 0000 0000 0000) {next_resync_marker () resync_marker 17 macroblock_number 1-12 quant_scale 5}} while (nextbits_bytealigned ()! = 000 0000 0000 0000 0000 0000 0000)}} else {if (background_composition) {load_backward_shape 1 if (load_backward ) {backward_shape_coding () load_foward_shape 1 if (load_foward_shape) foward_shape_coding ()}} ref_select_code 2 if (VOP_prediction_type == "01" || VOP_prediction_type == "10") {forward_temporal_ref 10 if (VOP_prediction_type = marker) backwaed_temporal_ref 10}} combined_motion_shape_texture_coding ()} next_state_code ()} The identifier (VOP_width) and (VOP_height) are the VO
This is a flag indicating the size of P.

The identifier (ref_select_code) is a flag indicating which layer image is used as a reference image based on the identifier (ref_layer_id) in forward (forward) and backward (backward) prediction. Details are shown in the table shown in FIG. 9 and the table shown in FIG.

FIG. 15 shows I and P pictures (image VO).
9 shows the syntax of a macroblock in P). (CO
D) is a flag indicating whether or not the data of the macro block exists thereafter. In the case of (COD = 1), it indicates that there is no data of the macro block thereafter (ie, a skip macro block). (COD = 0),
Further, a flag is transmitted. (MCBCP) is a flag indicating the type of the macroblock, and predetermined flags and data are transmitted according to the flag.

FIG. 16 shows the syntax of a macroblock in a B picture (VOP). If the corresponding macroblock of the most recently decoded I or P picture (picture VOP) is a skip macroblock (COD = 1), then that macroblock in the B picture (picture VOP) is also a skip macroblock. Become.

(MODB) is a flag indicating the type of macroblock in the B picture (VOP). FIG. 17 shows the variable length code of (MODB). If (MODB = 0), it indicates that there is no more macroblock data. (MODB =
In the case of 10), (CBPB) is not transmitted, and (MBTYPE) is transmitted. In the case of (MODB = 11), (CBPB) and (MBTYPE) are transmitted. Note that x in the drawing indicates the current macroblock.

(CBPB) is a 6-bit flag indicating whether or not a DCT coefficient exists in each block in the macroblock. If (CBPB) is not transmitted, (CBPB) is interpreted as 0, and no DCT coefficient is transmitted in the macroblock.

(MBTYPE) is a flag indicating the prediction mode of each macroblock in the B picture. Lower layer
(MBTYPE) is shown in FIG. As shown in FIG. 18, the flag transmitted in the macroblock is determined according to (MBTYPE). Note that x in the drawing indicates the current macroblock.

FIG. 19 shows (MBTYPE) in the upper layer. In the case of (MBTYPE = "1"), the two-layer prediction mode is set.

The conditions for the skip macroblock in the lower layer are as follows.

(A) P picture (VOP) (1) When COD = “1”.

In this case, the macro block is treated as a skip macro block. The DCT coefficients are all 0, and the motion vector is also treated as 0.

(B) B picture (VOP) (1) The most recently decoded I or P picture (VOP)
When the corresponding macroblock of P) is a skip macroblock (COD = 1).

In this case, a skip macro block is set. The prediction is performed in the same way as for a P picture (VOP), and the motion vector is treated as 0.

(2) In cases other than (1) and (MODB == '
0 ').

In this case, the macro block is treated as a skip macro block, and the (MBTYPE) of this macro block is set to H.264 of Direct. 263; H.263 is encoded in the same manner as the PB picture. At this time, the motion vector of the macroblock at the same position in the P picture (VOP) decoded immediately before is used.

The conditions for a skipped macroblock in the upper layer (scalability = 1) are as follows.

(A) P picture (VOP) (1) When COD == '1'

In this case, the macro block is treated as a skip macro block. The DCT numbers are all 0,
The motion vector is also treated as 0.

In the case of a P picture (VOP), the conditions for a skipped macroblock are the same as in the lower layer.

(B) Picture (VOP) In a skip macroblock, its prediction mode and reference image should be set to the most generally efficient mode. Therefore, it is more efficient to set the two-layer prediction mode to the macroblock of the upper layer in the case of spatial scalability.

In this embodiment, the upper layer B
The conditions of the skip macroblock of the picture (VOP) are as follows.

(1) When (ref_select_code == '00') and (MODB == '0').

In the case of (ref_select_code == '00'), regardless of whether or not the corresponding macroblock of the most recently decoded I or P picture (VOP) is a skip macroblock (COD value), The subsequent data is transmitted. At this time, if (MODB) to be encoded next is 0, it becomes a skipped macroblock, and no further data is transmitted.

At this time, the prediction is in the two-layer prediction mode,
The motion vector is treated as 0.

(2) (ref_select_code! = '00') and the most recently decoded I or P picture (V
OP) when the corresponding macroblock is a skip macroblock (COD = 1).

In this case, a skip macro block is set. The prediction is performed in the same way as for a P picture (VOP), and the motion vector is treated as 0.

(3) When (ref_select_code! = '00') and (MODB = = '0').

In this case, the macro block is treated as a skip macro block, and the (MBTYPE) of this macro block is set to H.264 of Direct (Direct). 263; H.263 is encoded in the same manner as the PB picture. At this time, the motion vector of the same macroblock in the P picture (VOP) decoded immediately before (VOL) indicated by (ref_select_code) is used.

[0218]

According to the present invention, the amount of change in the low-resolution image signal is obtained by using the motion information obtained from the high-resolution image signal, and this amount of change is added to the locally decoded image of the high-resolution image signal. By generating and encoding a predicted reference image, that is, by adding the amount of change of the lower layer to the reference image of the upper layer in addition to the prediction mode of the conventional MPEG system, a new reference image is generated. Generated, and the inter-frame predictive coding is realized using the new reference image. Accordingly, in the spatial scalable coding method, it is possible to improve prediction efficiency and improve coding efficiency.

[Brief description of the drawings]

FIG. 1 is a block circuit diagram showing a schematic configuration of an image signal encoding apparatus according to an embodiment for realizing an image signal encoding method and apparatus of the present invention.

FIG. 2 is a diagram used for explaining temporal scalability.

FIG. 3 is a block circuit diagram showing a specific configuration of a lower layer encoding circuit of the image signal encoding device.

FIG. 4 is a block circuit diagram illustrating a specific configuration of an upper layer encoding circuit of the image signal encoding device.

FIG. 5 is a diagram used to explain encoding in the case of spatial scalability.

FIG. 6 is a diagram used to explain encoding in the case of temporal scalability in the present embodiment.

FIG. 7 is a diagram used for describing a reference image of an upper layer according to the present embodiment.

FIG. 8 is a flowchart illustrating a flow of a method for generating a prediction reference image.

FIG. 9 shows a flag (ref_select_cod) in a P picture.
It is a figure showing the table of e).

FIG. 10 shows a flag (ref_select_cod) in a B picture.
It is a figure showing the table of e).

FIG. 11 is a block circuit diagram illustrating a schematic configuration of an image signal decoding device according to an embodiment for realizing the image signal decoding method and device of the present invention.

FIG. 12 is a block circuit diagram showing a specific configuration of a lower layer decoding circuit of the image signal decoding device.

FIG. 13 is a block circuit diagram showing a specific configuration of an upper layer decoding circuit of the image signal decoding device.

FIG. 14 is a diagram used for describing a hierarchical structure of video syntax.

FIG. 15 is a diagram illustrating the syntax of a macroblock in I and P pictures.

FIG. 16 is a diagram illustrating the syntax of a macroblock in a B picture.

FIG. 17 is a diagram illustrating a MODB variable length code which is a flag indicating a macroblock type in a B picture.

FIG. 18 is a diagram illustrating MBTYPE which is a flag indicating a prediction mode of each macroblock in a B picture of a lower layer.

FIG. 19 is a diagram illustrating MBTYPE in an upper layer.

FIG. 20 is a block circuit diagram illustrating a schematic configuration example of a conventional MP @ ML encoder of the MPEG system.

FIG. 21 is a block circuit diagram illustrating a schematic configuration example of a conventional decoder of MPEG MP @ ML.

FIG. 22 is a block circuit diagram illustrating a configuration example of a conventional encoder having spatial scalability.

FIG. 23 is a block circuit showing a configuration example of a conventional decoder of spatial scalability.

[Explanation of symbols]

1 image layering circuit, 2 delay circuit, 3 upper layer coding circuit, 4 resolution conversion circuit, 5 lower layer coding circuit, 6 multiplexing circuit, 81 multiplexing circuit,
82 delay circuit, 83 upper layer decoding circuit, 9
4 Resolution conversion circuit, 95 Lower layer decoding circuit

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N ^7/ 24-7/68

Claims (10)

(57) [Claims] An image signal encoding method for separating and hierarchizing an image signal into at least two layers of high-resolution and low-resolution image signals and encoding these high-resolution and low-resolution image signals using a predicted reference image. Detecting motion information by using the high-resolution image signal; obtaining a change amount of the low-resolution image signal by using the motion information; comparing the change amount with a predetermined threshold value; Adding the amount of change to the decoded image signal of the signal obtained by encoding the high-resolution image signal when the difference is equal to or larger than the threshold to generate a predicted reference image, and performing encoding using the generated predicted reference image. Characteristic image signal encoding method. 2. The image signal encoding method according to claim 1, wherein the amount of change is obtained from a signal in which the resolution of the low-resolution image signal matches the resolution of the high-resolution image signal. 3. The image signal according to claim 1, wherein the amount of change is compared with a predetermined threshold, and when the amount of change is less than the threshold, the value of the amount of change is set to zero and the addition is performed. Encoding method. 4. The image signal encoding method according to claim 3, wherein the comparison between the amount of change and the predetermined threshold is performed on a pixel-by-pixel basis. 5. An image signal coding apparatus for separating and layering an image signal into at least two layers of high-resolution and low-resolution image signals and coding these high-resolution and low-resolution image signals using a predicted reference image. A motion detecting means for detecting motion information using the high-resolution image signal; a change amount calculating means for obtaining a change amount of the low-resolution image signal using the motion information; and a predetermined threshold value for the change amount. Generating means for generating a predicted reference image by adding the amount of change to a decoded image signal of a signal obtained by encoding the high-resolution image signal when the amount of change is equal to or larger than the threshold, An encoding unit for performing encoding using the predicted reference image. 6. An image signal is separated and hierarchized into at least two layers of high-resolution and low-resolution image signals and encoded, and a bit stream comprising these high-resolution and low-resolution image signals and encoding parameters is predicted. In an image signal decoding method for decoding using a reference image, motion information is extracted from an encoding parameter, a change amount of the decoded low-resolution image signal is obtained using the motion information, and the change amount is determined by a predetermined threshold value. When the amount of change is equal to or greater than the threshold, the value of the amount of change is added to the decoded image signal of the high-resolution image signal to generate the predicted reference image, and the generated predicted reference image is used. An image signal decoding method characterized in that decoding is performed by performing decoding. 7. The image signal decoding method according to claim 6, wherein the amount of change is obtained from a signal in which the resolution of the decoded low-resolution image signal matches the resolution of the high-resolution image signal. 8. The method according to claim 1, wherein the change amount is compared with a predetermined threshold value, and when the change amount is less than the threshold value, the value of the change amount is set to zero and added to the high-resolution decoded image signal. The image signal decoding method according to claim 6. 9. The image signal decoding method according to claim 8, wherein the comparison between the amount of change and the predetermined threshold is performed on a pixel-by-pixel basis. 10. An image signal is separated and hierarchized into at least two layers of high-resolution and low-resolution image signals and encoded, and a bit stream including the high-resolution and low-resolution image signals and encoding parameters is predicted. An image signal decoding device that decodes using a reference image, extracting motion information from an encoding parameter, and using the motion information to determine a change amount of the decoded low-resolution image signal; Means for generating the predicted reference image by adding the value of the amount of change to the decoded image signal of the high-resolution image signal when the amount of change is equal to or greater than the threshold, Means for decoding using a reference image.