JPH10271512A - Image signal coder, its method, image signal decoder, its method and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance prediction efficiency and to improve coding efficiency in the space scalable coding method. SOLUTION: In the case of applying scalable coding to an image signal, a prediction image PV is configured by using a magnified image VM6 resulting from converting the resolution of a lower order layer image VOP 12 or a high order layer image VOP 13 in the unit of pixels in response to a value (difference image) after inverse discrete cosine transformation(IDCT) of a low-order layer macro block. The prediction coding mode in the unit of pixels is adaptively selected by using the prediction image PV configured as above to encode a high order layer image VOP 14.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image signal recorded on a recording medium such as a magneto-optical disk or a magnetic tape, reproduced and displayed on a display device, a video conference system, a video telephone system, and the like. , Broadcasting equipment, multimedia database search system, etc.
When a moving image signal is transmitted from a transmission side to a reception side via a transmission path, and the reception side receives and displays the signal,
The present invention relates to an image signal encoding device and an image signal encoding method suitable for use in editing and recording a moving image signal, an image signal decoding device and an image signal decoding method, and a recording medium.

[0002]

2. Description of the Related Art For example, in a system for transmitting a moving image signal to a remote place, such as a video conference system and a video telephone system, in order to efficiently use a transmission path, line correlation or inter-frame correlation of a video signal is required. Is used to compress and encode an image signal.

A typical high-efficiency moving picture coding method is the so-called MPEG (moving picture coding for storage) method. This is ISO-IEC / JTC1 / SC2 /
It is discussed at WG11 and proposed as a standard. Motion compensated predictive coding and DCT (Discrete Cosine Tr
ansform) A hybrid system combining coding is adopted. In MPEG, several profiles and levels are defined to support various applications and functions. The most basic is the main profile main level (MP @ ML).

[0004] Referring to FIG. 21, MPEG MP ＠
A configuration example of the ML encoder will be described.

An input image signal is first sent to a frame memory 20.
1 and thereafter read from the frame memory 201, sent to a subsequent circuit and encoded in a predetermined order.

That is, the image signal to be encoded is
The data is read from the frame memory 201 in macroblock units and input to the motion vector detection (ME) circuit 202. The motion vector detection circuit 202 processes an image signal of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance. Whether the image of each frame input sequentially is processed as one of I, P, and B pictures is predetermined (for example, I, B, P, B,
P,..., B, P).

Here, the motion vector detecting circuit 202
The motion vector is detected by referring to a predetermined reference frame and performing motion compensation. The motion compensation (inter-frame prediction) has three types of prediction modes: forward prediction, backward prediction, and bidirectional prediction. The prediction mode of a P picture is only forward prediction, and the prediction modes of a B picture are three types: forward prediction, backward prediction, and bidirectional prediction. The motion vector detection circuit 202 selects a prediction mode that minimizes a prediction error, and generates a motion vector at that time.

At this time, the prediction error is compared with, for example, the variance value of the macroblock to be coded. Will be in this case,
The prediction mode is intra-picture coding (intra). The motion vector and the prediction mode information are stored in a variable length coding (VLC) circuit 206 and a motion compensation (MC) circuit 21.
2 is input.

The motion compensation circuit 212 generates a predicted reference image signal based on a predetermined motion vector, and inputs the predicted reference image signal to the arithmetic circuit 203. Arithmetic circuit 203
Then, the frame memory 201 is divided into macroblock units.
And the value of the image signal to be encoded from
12 to obtain a difference signal from the value of the predicted reference image signal from
The difference signal is output to DCT circuit 204. Intra macroblock (macroblock that is intra-coded)
In this case, the arithmetic circuit 203 outputs the macroblock of the image signal to be coded to the DCT circuit 204 as it is.

In the DCT circuit 204, the operation circuit 20
3 or the image signal itself is DCT
(Discrete cosine transform) processing and conversion to DCT coefficients. This DCT coefficient is input to the quantization circuit 205,
Here, after being quantized in a quantization step corresponding to the amount of data stored in the transmission buffer 207 (the remaining amount of data that can be stored in the buffer), the variable length coding circuit 2
06.

The variable length coding circuit 206 performs quantization (Q)
According to the quantization step (quantization scale) supplied from the circuit 205, the quantized data supplied from the quantization circuit 205 is converted into a variable-length code such as a Huffman code, and the encoded data is transmitted. Output to the buffer 207.

The variable length coding circuit 206 also includes a quantization step (scale) from the quantization circuit 205 and a prediction mode (intra-picture prediction, forward prediction, backward prediction, or both directions) from the motion vector detection circuit 202. A prediction mode indicating which of the predictions has been set) and a motion vector are input, and these are also subjected to variable-length coding.

The transmission buffer 207 temporarily stores the input data, and outputs (buffer feedback) data corresponding to the storage amount to the quantization circuit 205 as a quantization control signal. That is, the transmission buffer 207
Is output from the quantization circuit 205 by increasing the quantization scale of the quantization circuit 205 by the quantization control signal when the data storage amount (remaining amount of storable data) increases to an allowable upper limit value. The data amount of the quantized data is reduced. Conversely, when the data storage amount (remaining amount of data that can be stored) decreases to the permissible lower limit, the transmission buffer 207 reduces the quantization scale of the quantization circuit 205 by using the quantization control signal. , The amount of quantized data output from the quantization circuit 205 is increased. In this way, overflow or underflow of the transmission buffer 207 is prevented.

[0014] The encoded data stored in the transmission buffer 207 is read at a predetermined timing and output as a bit stream to a transmission path.

On the other hand, the quantized data output from the quantization circuit 205 is also input to an inverse quantization (IQ) circuit 208. In the inverse quantization circuit 208, the quantization circuit 2
05 is inversely quantized corresponding to the quantization step also supplied from the quantization circuit 205. An output signal (DCT coefficient obtained by inverse quantization) of the inverse quantization circuit 208 is input to an IDCT (inverse DCT) circuit 209, where the output signal (image signal) is subjected to inverse DCT processing. Or a difference signal) is sent to the arithmetic circuit 210. In this arithmetic circuit 210,
If the output signal from the IDCT circuit 209 is a P-picture difference signal, the image signal is restored by adding the difference signal from the IDCT circuit 209 and the image signal from the motion compensation circuit 212. When the output signal from the IDCT circuit 209 is an intra macro block, the IDCT circuit 2
09 is output as it is. This image signal is stored in the frame memory 211. The motion compensation circuit 212 generates a predicted reference image signal using the image in the frame memory 211, the motion vector, and the prediction mode.

Next, with reference to FIG.
A configuration example of the ML decoder will be described.

The encoded image data (bit stream) transmitted via the transmission path is received by a receiving circuit (not shown) or reproduced by a reproducing device, and temporarily stored in a receiving buffer 221. The data is supplied to the variable length decoding (IVLC) circuit 222 as encoded data. The variable length decoding circuit 222 performs variable length decoding on the encoded data supplied from the reception buffer 221, and outputs the obtained motion vector and prediction mode to the motion compensation circuit 227, and the quantization step to the inverse quantization circuit 223. , And output each,
The variable-length decoded data (quantized data) is output to an inverse quantization (IQ) circuit 223.

The inverse quantization circuit 223 includes a variable length decoding circuit 2
The data (quantized data) supplied from 22 is inversely quantized according to the quantization step also supplied from the variable length decoding circuit 222, and the DCT coefficient obtained by inverse quantization is output to the IDCT circuit 224. . The DCT coefficient output from the inverse quantization circuit 223 is subjected to inverse DCT processing by the IDCT circuit 224, and the output signal (image signal or difference signal) is supplied to the arithmetic circuit 225.

If the output data from the IDCT circuit 224 is an I picture (image signal), the image signal is output from the arithmetic circuit 225 as it is, and a difference signal ( P or B
In order to generate a predicted reference image signal of (picture data), it is supplied to and stored in the frame memory 226. The image signal from the arithmetic circuit 225 is output to the outside as it is as a reproduced image.

On the other hand, if the input bit stream is P or B
In the case of a picture, the motion compensation circuit 227 generates a predicted reference image signal according to the motion vector and the prediction mode supplied from the variable length decoding circuit 222, and outputs the predicted reference image signal to the arithmetic circuit 225. The arithmetic circuit 225 adds the difference image signal input from the IDCT circuit 224 and the prediction reference image signal supplied from the motion compensation circuit 227, and outputs the result as a reproduced image. In the case of a P picture, the image signal from the arithmetic circuit 225 is also input to and stored in the frame memory 226, and is used as a reference image of an image signal to be decoded next.

In MPEG, various profiles and levels are defined in addition to MP @ ML, and various tools are prepared. The scalability described below is also MP
This is one of the EG's tools.

In MPEG, a scalable encoding method for realizing scalability corresponding to different image sizes and frame rates has been introduced. For example, in the case of spatial scalability, when decoding only the lower layer bit stream, an image signal with a small image size is decoded, and when decoding the lower layer and upper layer bit streams, an image signal with a large image size is decoded. I do.

An encoder for spatial scalability will be described with reference to FIG. In the case of spatial scalability, the lower layer corresponds to an image signal with a small image size, and the upper layer corresponds to an image signal with a large image size.

The image signal of the lower layer is first input to the frame memory 261, and the above-described M
Encoded in the same way as P @ ML.

That is, data read from the frame memory 261 in macroblock units is input to the motion vector detection circuit 262. The motion vector detection circuit 262 processes the image data of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance.

The motion vector detection circuit 262 detects a motion vector by referring to a predetermined reference frame (ie, a forward original image, a backward original image, and an original image) and performing motion compensation. The motion compensation (inter-frame prediction) has three types of prediction modes: forward prediction, backward prediction, and bidirectional prediction. The motion vector detection circuit 262
A prediction mode that minimizes a prediction error is selected, and a motion vector at that time is generated. The information on the motion vector and the prediction mode is input to the variable length coding circuit 266 and the motion compensation circuit 272.

The motion compensation circuit 272 generates a predicted reference image signal based on a predetermined motion vector, and inputs the predicted reference image signal to the arithmetic circuit 263. Arithmetic circuit 263
In the above, the frame memory 261 is used for each macro block.
And the value of the image signal to be encoded from
A difference signal from the value of the predicted reference image signal from 72 is obtained,
The difference signal is output to DCT circuit 264. In the case of an intra macroblock (a macroblock to be intra-coded), the arithmetic circuit 263 outputs the macroblock signal to be encoded to the DCT circuit 264 as it is.

In the DCT circuit 264, the operation circuit 26
3 is subjected to DCT processing and converted into DCT coefficients. The DCT coefficient is input to the quantization circuit 265, where it is quantized in a quantization step corresponding to the amount of data stored in the transmission buffer 267 (the remaining amount of data that can be stored in the buffer), and is then changed as quantized data. It is input to the long encoding circuit 266.

The variable length coding circuit 266 is
Quantization step (quantization scale) supplied from 65
, The quantized data supplied from the quantizing circuit 265 is converted into a variable-length code such as a Huffman code, and the coded data is output to the transmission buffer 267.

The variable length coding circuit 266 also includes a quantization step (scale) from the quantization circuit 265 and a prediction mode (intra-picture prediction, forward prediction, backward prediction, or bidirectional prediction) from the motion vector detection circuit 262. A prediction mode indicating which of the predictions has been set) and a motion vector are input, and these are also subjected to variable-length coding.

The transmission buffer 267 temporarily stores the input coded data, and outputs (buffer feedback) data corresponding to the stored amount to the quantization circuit 265 as a quantization control signal. This prevents the transmission buffer 267 from overflowing or underflowing.

The encoded data stored in the transmission buffer 267 is read out at a predetermined timing, and is output to a transmission path as a bit stream.

On the other hand, the quantized data output from the quantization circuit 265 is also input to the inverse quantization circuit 268. In the inverse quantization circuit 268, the quantized data supplied from the quantization circuit 265 is inversely quantized in accordance with the quantization step also supplied from the quantization circuit 265.
The output signal (DCT coefficient) of the inverse quantization circuit 268 is
After being input to the IDCT circuit 269 and subjected to inverse DCT processing here, the output signal (image signal or difference signal) is sent to the arithmetic circuit 270. In this arithmetic circuit 270,
When the output signal from the IDCT circuit 209 is a P-picture difference signal, the image signal is restored by adding the difference signal from the IDCT circuit 269 and the image signal from the motion compensation circuit 272. When the output signal from the IDCT circuit 209 is an intra macro block, the image signal from the IDCT circuit 209 is output as it is. This image signal is stored in the frame memory 271. The motion compensation circuit 272 generates a prediction reference image signal using the image signal, the motion vector, and the prediction mode of the frame memory 271.

However, in the configuration example of the lower layer, the output image signal of the arithmetic circuit 270 is not only supplied to the frame memory 271 and used as a reference image of the lower layer, but also an image to be enlarged by upsampling. After being enlarged by the enlargement circuit 243 to the same image size as the image size of the upper layer, it is also used for the reference image of the upper layer.

That is, the image signal from the arithmetic circuit 270 is input to the frame memory 271 and the image enlargement circuit 243 as described above. The image enlargement circuit 243 enlarges the image signal generated by the arithmetic circuit 270 to have the same size as the image size of the upper layer, and the weighting circuit 2
44.

The weighting circuit 244 includes an image enlargement circuit 2
43 is multiplied by the weight (1-W) and output to the arithmetic circuit 258.

On the other hand, the image signal of the upper layer is first input to the frame memory 245. The motion vector detection circuit 246 determines a motion vector and a prediction mode, similarly to the above-described MP @ ML.

Here, in the configuration of the upper layer,
The motion compensation circuit 256 generates a predicted reference image signal according to the motion vector and the prediction mode determined by the motion vector detection circuit 246. The predicted reference image signal is supplied to the weighting circuit 257. In the weighting circuit 257, the weight W
(Weight coefficient W) and outputs the result to the arithmetic circuit 258.

The arithmetic circuit 258 adds the image signals from the weighting circuits 244 and 257, and outputs the obtained image signal to the arithmetic circuit 247 as a predicted reference image signal.
The image signal from the arithmetic circuit 258 is also transmitted to the arithmetic circuit 254.
The image signal is also input to the frame memory 255, is added to the image signal from the IDCT circuit 253, and is used as a reference image signal of an image signal to be input and thereafter encoded.

The arithmetic circuit 247 includes a frame memory 245
, And the difference between the image signal to be encoded and the predicted reference image signal from the arithmetic circuit 258 is calculated and output. However,
In the case of the intra-coded macroblock, the arithmetic circuit 24
Reference numeral 7 denotes a DCT circuit 248 for converting the image signal to be encoded as it is.
Output to

The DCT circuit 248 performs a DCT (Discrete Cosine Transform) process on the output signal of the arithmetic circuit 247, generates a DCT coefficient, and outputs the DCT coefficient to the quantization circuit 249. Quantization circuit 2
49, the transmission buffer 251 is the same as in the case of MP @ ML.
The DCT coefficient is quantized according to a quantization scale determined based on the data storage amount of the DCT coefficient, and is output to the variable length encoding circuit 250 as quantized data. The variable length coding circuit 250 performs variable length coding on the quantized data, and then outputs the coded data as a bit stream of an upper layer via the transmission buffer 251.

The quantized data from the quantization circuit 249 is inversely quantized by the inverse quantization circuit 252 using the quantization scale used in the quantization circuit 249, and the output data (DCT coefficient) is transmitted to the IDCT circuit 253. Output. Further, after the IDCT circuit 253 performs the inverse DCT processing on the DCT coefficient, an output signal (image signal or difference signal) is input to the arithmetic circuit 254. When the output signal from the IDCT circuit 253 is a P-picture differential image, the arithmetic circuit 254 determines whether the image signal from the arithmetic circuit 258 and the IDCT circuit 2
The difference signal from the signal 53 is added to restore the image signal. When the output signal from the IDCT circuit 253 is an intra macro block, the image signal from the IDCT circuit 253 is output as it is. This image signal is stored in the frame memory 255. The motion compensation circuit 256 generates a predicted reference image signal using the image signal, the motion vector, and the prediction mode of the frame memory 255.

The variable length coding circuit 250 also includes the motion vector and the prediction mode detected by the motion vector detection circuit 246, the quantization scale used by the quantization circuit 249, and the weights used by the weighting circuits 244 and 257. W
Are input, respectively encoded and transmitted.

Next, an example of a spatial scalability decoder will be described with reference to FIG.

After the lower layer bit stream is input to the reception buffer 301, it is decoded in the same manner as MP @ ML. That is, the encoded data read from the reception buffer 301 is sent to the variable length decoding circuit 302.
The variable length decoding circuit 302 performs variable length decoding on the encoded data supplied from the reception buffer 301, and outputs the motion vector and the prediction mode to the motion compensation circuit 307, and outputs the quantization step to the inverse quantization circuit 303, respectively. At the same time, the variable-length decoded data (quantized data) is output to the inverse quantization circuit 303 in macroblock units.

The inverse quantization circuit 303 includes a variable length decoding circuit 3
02 (quantized data) is inversely quantized in accordance with the quantization step also supplied from the variable length decoding circuit 302, and the DCT coefficient obtained by inverse quantization is output to the IDCT circuit 304. . The DCT coefficient output from the inverse quantization circuit 303 is subjected to inverse DCT processing in the IDCT circuit 304, and an output signal (image signal or difference signal) is supplied to the arithmetic circuit 305.

If the output signal from the IDCT circuit 304 is I-picture data, the image signal is output from the arithmetic circuit 305 as it is, and the differential signal (P or B picture The data is supplied to and stored in the frame memory 306 to generate a predicted reference image signal of (data). This image signal is output to the outside as a reproduced image as it is.

On the other hand, if the input bit stream is P or B
In the case of a picture, the motion compensation circuit 307 generates a predicted reference image signal according to the motion vector and the prediction mode supplied from the variable length decoding circuit 302, and outputs the predicted reference image signal to the arithmetic circuit 305. The arithmetic circuit 305 adds the difference signal input from the IDCT circuit 304 and the predicted reference image signal supplied from the motion compensation circuit 307, and outputs the result as an image signal. In the case of a P picture, the image signal from the arithmetic circuit 305 is also input to and stored in the frame memory 306, and is used as a predicted reference image signal of an image signal to be decoded next.

However, in the configuration of FIG. 24, the image signal from the arithmetic circuit 305 is output to the outside as described above,
Further, the image data is stored in the frame memory 306 and is not only used as a prediction reference image signal of an image signal to be decoded thereafter, but is also enlarged by the image enlargement circuit 327 to the same image size as the image signal of the upper layer. Are also used as predicted reference image signals.

That is, the image signal of the arithmetic circuit 305 is
As described above, it is output as a reproduced image signal of the lower layer, and is output to the frame memory 306.
After being supplied to the image enlargement circuit 327 and enlarged to the same image size as the image size of the upper layer, the weight addition circuit 3
28.

In the weighting circuit 328, the image enlargement circuit 3
The 27 image signals are multiplied by (1-W) calculated using the decoded weight W, and this value is calculated by the arithmetic circuit 317.
Output to

On the other hand, the bit stream of the upper layer is
The data is supplied to the variable length decoding circuit 310 via the reception buffer 309, where the encoded data is variable length decoded, and the quantization scale, motion vector, prediction mode, and weight coefficient are decoded together with the quantized data. Variable length decoding circuit 31
The quantized data variable-length-decoded by 0 is inversely quantized by the inverse quantization circuit 311 using the decoded quantization scale, and then output to the IDCT circuit 312 as a DCT coefficient. The IDCT circuit 312 performs an inverse DCT process on the DCT coefficient, and an output signal (image signal or difference signal) is supplied to the arithmetic circuit 313.

The motion compensation circuit 315 generates a predicted reference image signal according to the decoded motion vector and the prediction mode, and applies the predicted reference image signal to the weighting circuit 3.
Enter 16 The weighting circuit 316 multiplies the decoded reference image signal from the motion compensation circuit 315 by the decoded weight W, and outputs an image signal obtained from the multiplication result to the arithmetic circuit 317.

The arithmetic circuit 317 adds the image signals of the weighting circuits 328 and 316 and outputs the obtained image signal to the arithmetic circuit 313. In the arithmetic circuit 313, the IDC
When the output signal from the T circuit 312 is a difference signal,
The difference signal from the IDCT circuit 312 and the image signal from the arithmetic circuit 317 are added to restore the upper layer image signal. When the output signal from the IDCT circuit 312 is an intra macro block, the image signal from the IDCT circuit 312 is output as it is. This image signal is stored in the frame memory 314, and is thereafter used as a predicted reference image signal of the image signal to be decoded.

Although the above description is for the processing of a luminance signal, the processing of a chrominance signal is similarly performed. However, in this case, a motion vector obtained by halving the luminance vector for the luminance signal in the vertical and horizontal directions is used.

The MPEG system has been described above.
In addition, various high-efficiency coding schemes for moving images have been standardized. For example, a so-called ITU-T (Internationa
l Telecommunication Union-Telecommunication secto
r: Telecommunications Standardization Sector of the International Telecommunications Union), mainly uses H.264 as an encoding method for communications. 261 and H.E. 263 is defined. This H. 261 and H.E. 263
Is basically a combination of motion-compensated predictive coding and DCT transform coding as in the MPEG system. Although details such as header information are different, the coding apparatus and the decoding apparatus are the same.

[0057]

In MPEG2, spatial scalability has already been standardized, but its coding efficiency is hardly sufficient. Therefore, M
In the PEG4 system and other new coding systems, it is an issue to improve the coding efficiency of spatial scalability.

The spatial scalability in the MPEG2 system will now be described in some detail. In the scalable coding method, the lower layer is coded in the same manner as a normal coding method, that is, MP @ ML in the case of MPEG2. The upper layer uses an image of the lower layer at the same time and an image decoded immediately before the same layer as a reference image. At this time, the prediction modes of the lower layer and the upper layer are determined completely independently. Therefore,
Despite the information being transmitted in the lower layer, there is a case where the information is not used at all in the upper layer, but is predicted from the decoded image in the upper layer and encoded.
This is equivalent to transmitting information that can be shared between the upper layer and the lower layer completely independently.

Therefore, it is necessary to reduce the duplication of information transmission as described above and to improve the coding efficiency.

In the MPEG2 system, the coding mode can be specified only in macroblock units. This is fine when dealing with images in relatively uniform areas,
When one macroblock contains a sequence that moves in a complicated manner or an image having a different property (for example, a still area and a moving area), it causes a reduction in coding efficiency.

Accordingly, the present invention has been made in view of such circumstances, and in a spatial scalable encoding method, an image signal encoding method capable of improving prediction efficiency and improving encoding efficiency. It is an object to provide a decoding device and method, an image signal decoding device and method, and a recording medium.

[0062]

SUMMARY OF THE INVENTION An image signal encoding apparatus according to the present invention has been proposed to solve the above-mentioned problems, and includes a lower-layer image signal and an upper-layer image signal respectively representing a predetermined image signal. An apparatus for encoding an image signal,
A first encoding unit that encodes a lower layer image signal using a reference image signal and outputs first encoded data, and a second encoder that encodes an upper layer image signal using a reference image signal and outputs a second encoded image signal. A second encoding unit that outputs encoded data;
A first decoding unit that decodes the encoded data of (a) to generate a first reference image signal, and a second decoding unit that decodes the second encoded data to generate a second reference image signal. Have
The second encoding means encodes using a third reference image signal generated by adaptively switching the first and second reference image signals for each pixel.

The image signal encoding method of the present invention has been proposed in order to solve the above-mentioned problem, and encodes a lower-layer image signal and an upper-layer image signal respectively representing a predetermined image signal. A first encoding step of encoding a lower layer image signal using a reference image signal and outputting first encoded data, and using an upper layer image signal using a reference image signal. A second encoding step of encoding and outputting the second encoded data; a first decoding step of decoding the first encoded data to generate a first reference image signal; A second decoding step of decoding the encoded data to generate a second reference image signal, wherein the second encoding step applies the first and second reference image signals to each pixel. Third reference generated by dynamic switching It is characterized by encoding using an image signal.

Next, an image signal decoding apparatus according to the present invention has been proposed to solve the above-mentioned problem, and comprises an encoded lower layer image signal and an encoded upper layer image signal. Receiving encoded data, and decoding the encoded data, wherein the encoded lower layer image signal and the encoded upper layer image signal are each encoded using a reference image signal. Receiving means for receiving encoded data, first decoding means for decoding the encoded lower-layer image signal using the reference image signal, and outputting the decoded lower-layer image signal. Decoding means for decoding the coded image signal of the upper layer using the reference image signal and outputting the decoded image signal of the upper layer, the decoded lower-layer image signal Is the first reference image signal The used and decoded upper layer image signal is used as a second reference image signal, and the second decoding unit generates the image signal by adaptively switching the first and second reference image signals for each pixel. The decoding is performed using the obtained third reference image signal.

The image signal decoding method of the present invention has been proposed in order to solve the above-mentioned problem, and comprises an encoded lower layer image signal and an encoded upper layer image signal. A method for receiving encoded data and decoding the encoded data, wherein the encoded lower-layer image signal and the encoded upper-layer image signal are each encoded using a reference image signal. Receiving a coded data, a first decoding step of decoding the coded lower layer image signal using the reference image signal and outputting the decoded lower layer image signal, A second decoding step of decoding the coded upper layer image signal using the reference image signal and outputting the decoded upper layer image signal, wherein the decoded lower layer image signal is First reference The image signal of the upper layer decoded and used as an image signal is used as a second reference image signal. In the second decoding step, the first and second reference image signals are adaptively switched for each pixel. The decoding is performed using the third reference image signal generated as described above.

Next, the recording medium of the present invention is a recording medium that can be decoded by an image signal decoding device, and is composed of an encoded lower layer image signal and an encoded upper layer image signal. A first encoding step of encoding a lower-layer image signal using a reference image signal and outputting first encoded data; and an upper-layer image. A second encoding step of encoding the signal using the reference image signal and outputting second encoded data, and a first encoding step of decoding the first encoded data to generate a first reference image signal A decoding step, and a first decoding step of decoding the second encoded data to generate a second reference image signal, wherein the second encoding step includes:
It is characterized by being encoded using a third reference image signal generated by adaptively switching the first and second reference image signals for each pixel.

[0067]

Embodiments of the present invention will be described below with reference to the drawings.

First, before giving a detailed description of the embodiment of the present invention, the basic concept of the present invention will be described in connection with the properties of each reference image in MPEG. Normally, the image signal of the upper layer has better image quality than the image signal of the lower layer. Therefore, when the change between frames is small, it is often the case that the motion compensation is performed using the image of the upper layer to reduce the prediction error. However, when the change between frames is large, the efficiency of motion compensation is reduced. Therefore, it is more efficient to use an image at the same time as a reference image. Therefore, in an area where the difference between frames is large, an image signal from a lower layer at the same time is
When the difference between frames is small, an image of the same layer (upper layer) may be used as a reference image.

Next, the duplication of the information of the spatial scalable coding of the MPEG system will be described.

Consider a case where a macroblock having an upper layer is encoded. At this time, when information (DCT coefficient) is transmitted in a macroblock of a lower layer corresponding to the macroblock of the upper layer, if prediction is performed with reference to the same image of the upper layer, the lower layer and the upper layer will be different. Information will be transmitted independently. That is, the information transmitted in the lower layer is not used for encoding in the upper layer, which causes a reduction in efficiency. Therefore,
When the DCT coefficients are transmitted in the corresponding lower layer macroblock, the use of the lower layer image as the prediction reference image reduces the duplication of data transmission and is more efficient.

As described above, (1) when coding a macroblock having an upper layer, in a region where the difference between frames is large, it is better to use the image at the same time (image of the lower layer) as a reference image. (2) When encoding a macroblock having an upper layer, if the DCT coefficient of the corresponding lower layer macroblock is transmitted, it is better to use the image of the lower layer as a reference image. Can be said.

In a conventional image signal encoding system such as MPEG2, the prediction mode is switched in units of macro blocks. In the case of a uniform image area, such prediction mode switching in macroblock units is sufficient. However, if the image moves in a complicated manner or the size of the image is small, the area where the image changes temporally and the area where the image does not change in the macroblock are mixed, causing a decrease in efficiency. It has become.

Therefore, if the prediction mode can be switched on a pixel basis, the coding efficiency can be improved.

From the above, in the present invention, I
Referring to the difference image after DCT, a pixel whose difference value is equal to or less than a certain threshold value uses a pixel of an upper layer, and a pixel value whose difference value exceeds a certain threshold value uses a pixel of a lower layer. Is generated to generate a reference image used to generate the reference image. At this time, the generated reference image is used by switching the image of the upper layer and the lower layer in pixel units. In the present invention, the IDCT
By performing pixel-by-pixel switching using the later difference image, it is possible to cope with parallel movement.

That is, in the present invention, the reference image generated by adaptively switching the pixel unit, the image of the lower layer after resolution conversion (in this embodiment, after image enlargement), and the image immediately before the upper layer. The decoded image is used as a reference image. Also, at this time, by switching the image that minimizes the prediction error in units of macroblocks, determining a prediction reference image, performing motion compensation, and encoding,
It is possible to improve the coding efficiency of spatial scalable coding and improve the image quality.

FIG. 1 shows an example of an image signal encoding apparatus according to an embodiment of the present invention.

In FIG. 1, an input image signal and a key signal are first input to an image signal layering circuit 1.

The input image signal is an image representing one image when a certain image signal is separated into a plurality of image signals (background image and moving object image) and encoded for each image. Signal. The key signal is a signal indicating contour information of the separated moving object image. For example, the key signal is a binary hard key or a multi-value soft key. further,
A flag (VOP offset) indicating the position of the separated moving object image in the absolute coordinates of the background image and a flag (VOP size) indicating the size of the separated moving object image are supplied to the image signal layering circuit 1.

The image signal layering circuit 1 separates an input image signal and a key signal into a plurality of layers. Although FIG. 1 shows a configuration in the case of two layers (one lower layer and one upper layer), it can be similarly divided into a plurality of layers. For simplicity, here 2
The case of separating into two layers will be described hereinafter.

For example, in the case of spatial scalability, the image signal layering circuit 1 converts the resolution of an input image signal and a key signal to generate an image signal and a key signal of a lower layer and an upper layer.

In the case of temporal scalability (scalability in the time axis direction), for example, the image signal layering circuit 1 switches the output of the image signal and the key signal to the lower layer and the upper layer according to the time and outputs. For example, in the case of FIG. 2, the sequentially supplied images (pictures) VOP0 to VOP6 are images VOP0, VOP2,
VOP4 and VOP6 are in the lower layer and the image VOP
1, VOP3 and VOP5 are output to the upper layer. In the case of the temporal scalability, resolution conversion such as enlargement or reduction of an image signal is not performed unlike the spatial scalability described above.

Further, for example, so-called SNR (Signal to
In the case of Noise Retio) scalability, the image signal layering circuit 1 outputs the input image signal and the key signal as they are to each layer. That is, the same image signal and key signal are output to the lower layer and the upper layer.

In the present embodiment, the case of the above spatial scalability is taken as an example.

For example, in the case of spatial scalability, the image signal layering circuit 1 outputs an image signal and a key signal obtained by reducing (resolution conversion) the input image signal and the key signal as a lower layer, while inputting it to an upper layer. The image signal and the key signal are output as they are. The resolution conversion here is, for example, a reduction filtering process using a thinning filter. The image signal layering circuit 1 may output an image signal and a key signal obtained by enlarging (resolution converting) the input image signal as an upper layer, and may output the input image signal and the key signal as they are to a lower layer. . The resolution conversion in this case is an enlargement filtering process using an enlargement filter or the like. Further, the image signal layering circuit 1 outputs two independently generated image signals and a key signal (which may have different resolutions or the same resolution) to an upper layer and a lower layer, respectively. Is also good. In this case, which image signal and key signal are output to the upper layer and the lower layer
It is predetermined. The image signal and the key signal of the upper layer are sent to the upper layer encoding circuit 3 via the delay circuit 2, and the image signal of the lower layer is transmitted to the lower layer encoding circuit 5.
Sent to

As described above, the image signal layering circuit 1 that converts the resolution of the input image signal and the key signal and outputs the converted signal to the lower layer and the upper layer also increases the magnification of the resolution of the upper layer image with respect to the lower layer image. A flag FR is also output. The flag FR is sent to the resolution conversion circuit 4 and the upper layer encoding circuit 3 via the delay circuit 2. Note that the resolution conversion circuit 4 is different from the resolution conversion means provided in the image signal layering circuit 1. The image signal layering circuit 1 further includes a flag (lower layer VOP size) indicating the size of the lower layer image (VOP), a flag (lower layer VOP offset) indicating a position in absolute coordinates, and a higher layer Flag (upper layer VO) indicating the size of the image (VOP)
A flag indicating the position in the absolute coordinates (upper layer VOP offset) is output to the lower layer encoding circuit 3 and the upper layer encoding circuit 5 for supply to the lower layer encoding circuit 3 and the upper layer encoding circuit 5, respectively. These flags are encoded by the upper layer encoding circuit 3 and the lower layer encoding circuit 5,
Furthermore, it is encoded as a bit stream, supplied to the scalable decoding device, and used in the scalable decoding device.

The encoding process and the decoding process in each layer will be described below. The input image signal of the upper layer, the input image signal of the lower layer, the input key signal of the upper layer, and the input key signal of the lower layer will be described. Since the encoding process and the decoding process of the signal are the same, only the encoding process and the decoding process of the input image signal of the upper layer and the input image signal of the lower layer will be described below, and the key signal of the upper layer and the lower layer will be described. The details of the encoding process and the decoding process of the key signal are omitted.

Here, a specific configuration of the lower layer coding circuit 5 will be described with reference to FIG.

In FIG. 3, the input image signal of the lower layer supplied to the lower layer coding circuit 5 is first input to the frame memory 21, read out in a predetermined order, and coded by the configuration of the subsequent stage. . The image data to be encoded is stored in a frame memory 21 in macroblock units.
And input to the motion vector detection circuit 22. The motion vector detection circuit 22 processes the image data of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance. Whether the image of each frame input sequentially is processed as one of I, P, and B pictures is predetermined (for example, I, B, P, B,
P,..., B, P).

Here, the motion vector detection circuit 22 refers to a predetermined reference frame (a forward original image, a backward original image, and an original frame image) and performs motion compensation, thereby obtaining the motion vector. To detect. Motion compensation (inter-frame prediction) has three types of prediction modes: forward prediction, backward prediction, and bidirectional prediction. The prediction mode of a P picture is only forward prediction, and the prediction modes of a B picture are three types: forward prediction, backward prediction, and bidirectional prediction. The motion vector detection circuit 22 selects a prediction mode that minimizes a prediction error, and generates a prediction vector at that time.

At this time, the prediction error is compared with, for example, the variance value of the macroblock to be coded. If the variance value of the macroblock is smaller, the prediction is not performed on the macroblock and the intra-frame coding is not performed. Done. In this case, the prediction mode is intra-picture encoding (intra). The motion vector and the information on the prediction mode are input to the variable length coding circuit 26 and the motion compensation circuit 32.

The motion vector is also supplied to an upper layer image signal encoding circuit, that is, the upper layer encoding circuit 3 in FIG.

The motion compensation circuit 32 generates a predicted reference image signal based on a predetermined motion vector, and supplies the predicted reference image signal to the arithmetic circuit 23. The arithmetic circuit 23 calculates the difference between the value of the image signal to be coded from the frame memory 21 and the value of the predicted reference image signal from the motion compensation circuit 32 in macroblock units. Output to In the case of an intra macro block, the arithmetic circuit 23 outputs the image signal to be encoded to the DCT circuit 24 as it is. The motion vector detection circuit 22 and the motion compensation circuit 32 are supplied with the lower layer VOP size and the lower layer VOP offset, and are used by the respective circuits. However, they are not shown in order to avoid complicating the drawings.

The DCT circuit 24 converts the difference signal into a DC signal.
T (Discrete Cosine Transform) processing is performed to convert to DCT coefficients. The DCT coefficient is input to the quantization circuit 25 and is quantized in a quantization step corresponding to the amount of data stored in the transmission buffer 27 (the remaining amount of data that can be stored in the buffer). Input to the conversion circuit 26.

The variable length coding circuit 26 is
In response to the supplied quantization step (quantization scale), the quantized data supplied from the quantization circuit 25 is converted into a variable length code such as a Huffman code, for example.
This encoded data is output to the transmission buffer 27.

The variable length coding circuit 26 also includes a quantization step (quantization scale) from the quantization circuit 25 and a prediction mode (intra-picture prediction, forward prediction, backward prediction, or prediction mode) from the motion vector detection circuit 22. A mode indicating which of the bidirectional predictions has been set), a motion vector, a flag indicating the size of the image (VOP) of the lower layer (lower layer VOP size), and a flag indicating the position in absolute coordinates (lower layer VOP) Offset), which are also variable-length coded.

[0096] Although details will be described later, when there is no macroblock data, the macroblock is treated as a skipped macroblock. That is, when the value of the motion vector is 0 and the quantized DCT coefficient is 0. Then, a flag COD indicating whether macroblock data exists is transmitted. If data of a macroblock exists, COD = 0, and if no data of a macroblock exists (skip macroblock), COD = 1. This flag is also supplied to the upper layer coding circuit 3.

The transmission buffer 27 temporarily stores the input coded data, and feeds back the coded data corresponding to the stored amount to the quantization circuit 25 as a quantization control signal. That is, when the data storage amount (remaining amount of data that can be stored) increases to the allowable upper limit, the transmission buffer 27 increases the quantization scale of the quantization circuit 25 by the above-described quantization control signal. 25, the data amount of the quantized data output from the control unit 25 is reduced. Conversely, when the data storage amount (remaining amount of data that can be stored) decreases to the allowable lower limit, the transmission buffer 27 reduces the quantization scale of the quantization circuit 25 by the quantization control signal. , Quantization circuit 25
Increase the data amount of the quantized data output from.
In this way, overflow or underflow of the transmission buffer 27 is prevented.

The encoded data stored in the transmission buffer 27 is read out at a predetermined timing and output to the transmission path as a lower layer bit stream.

On the other hand, the quantized data output from the quantization circuit 25 is also input to the inverse quantization circuit 28. In the inverse quantization circuit 28, the quantized data supplied from the quantization circuit 25 is inversely quantized according to the quantization step also supplied from the quantization circuit 25. The output signal of this inverse quantization circuit 28 (DCT coefficient obtained by inverse quantization) is input to an IDCT (inverse DCT) circuit 29, where it is subjected to inverse DCT processing, and then the output signal (image signal or The difference signal is sent to the arithmetic circuit 30. When the output signal of the IDCT circuit 29 is a difference signal, the arithmetic circuit 30 restores the image signal by adding the image signal from the motion compensation circuit 32 and the difference signal from the IDCT circuit 29. When the output signal of the IDCT circuit 29 is an intra macro block, the image signal from the IDCT circuit 29 is output as it is. This image signal is
The data is supplied to and stored in the frame memory 31. The motion compensation circuit 32 generates a predicted reference image signal using the image in the frame memory 31, the motion vector, and the prediction mode.

The output signal of the IDCT circuit 29, that is, the locally decoded difference signal is also supplied to the upper layer encoding circuit 3 of FIG.

The frame memory 31 reads out a predetermined locally decoded image and outputs it to the resolution conversion circuit 4 in FIG. 1 in accordance with the coding in the upper layer coding circuit 3 in FIG.

Returning to FIG. 1, the resolution conversion circuit 4 determines the resolution of the image signal supplied from the lower layer encoding circuit 5 according to the flag FR indicating the magnification of the resolution of the upper layer image with respect to the lower layer image. , And is supplied to the upper layer encoding circuit 3 by filtering processing (enlargement processing in this case). When the magnification is 1, that is, when the size of the upper layer is equal to the size of the lower layer, the resolution conversion circuit 4 outputs the image signal without any operation.

The upper layer image signal generated by the image signal layering circuit 1 is supplied to an upper layer encoding circuit 3 via a delay circuit 2. The delay circuit 2 delays the image signal of the upper layer by a time required for encoding the image signal of the predetermined lower layer in the lower layer encoding circuit 5.

Next, a specific configuration of the upper layer coding circuit 3 will be described with reference to FIG.

In FIG. 4, the input image signal of the upper layer supplied to the upper layer coding 3 is first input to and stored in the frame memory 41, read out in a predetermined order, and encoded by the subsequent stage. Be transformed into The image data to be encoded is read from the frame memory 41 in macroblock units, and input to the motion vector detection circuit 42. The motion vector detection circuit 42 processes the image data of each frame as an I picture, a P picture, or a B picture according to a predetermined sequence set in advance. It is determined in advance whether an image of each sequentially input frame is processed as a picture of I, P, or B (for example, I, P, or B).
B, P, B, P,... B, P are processed in this order).

Here, in the case of spatial scalability, the image signals of the upper layer and the lower layer are coded as shown in FIG. 5, for example. 5, first image VOP21 _L of the lower layer is encoded as an I picture. Also, the image VOP2 of the second and lower layers below
1 _{L to} VOP 24 _L are encoded as P pictures.
The reference image at this time is encoded using the image of the immediately preceding lower layer as the reference image. Further, the first image VOP21 _M of the upper layer is encoded as a P picture. The reference image at this time is an image V of the lower layer at the same time.
OP21 _L is obtained. The image VO of the second or higher layer
P22 _M ~VOP24 _M is encoded as B pictures. In this case, these images VOP22 _M ~VOP24 _M
Are the images of the immediately preceding upper layer and the lower layer images VOP22 _L , VOP23 _L ,
It encoded using VOP24 _L as the reference image. Also, in this case, the B picture of the upper layer becomes a reference image when encoding another image VOP, like the P picture of the lower layer. Note that SNR scalability is a special case of spatial scalability, in which the size of the upper layer is equal to that of the lower layer, and the coding procedure is the same.

In the case of temporal scalability, encoding is performed, for example, as shown in FIG. In FIG. 2, the first image VOP0 of the lower layer
Is encoded as an I picture. Also, the images VOP2 to VOP6 of the second and lower layers are encoded as P pictures. The reference image at this time is encoded using the image of the immediately preceding lower layer as the reference image. Also, the image VOP1 of the upper layer is encoded as a B picture, and the image VOP1 is the image VOP of the lower layer.
OP0 and the image VOP2 are set as reference images. In the case of the image VOP3 of the upper layer, it is encoded as a B picture, and the image VOP1 of the immediately preceding upper layer and the image VOP4 of the lower layer are used as reference images. In the case of the upper layer image VOP5, it is encoded as a P picture, and the immediately preceding upper layer image VOP3 is used as a reference image.

Hereinafter, reference pictures of P and B pictures in the upper layer will be described.

At the time of prediction of an upper layer, not only an image of the same layer but also an image of another layer (scalable layer) can be used as a reference image. For example, in the case of scalability of two layers as in the present embodiment, the upper layer can use the image of the lower layer as a reference image.

The P picture performs forward prediction, and an image of the same layer or an image of another layer can be used as a reference image. In a B picture, an image of the same layer or an image of another layer can be used as a reference image for forward prediction and backward prediction. Also, in the case of a B picture, in addition to the two reference images of the front and rear,
It is also possible to use a third image generated from these two reference images as a reference image used for generating a predicted reference image.

This will be described with reference to FIG. Image VOP
11, VOP12 indicates an image of a lower layer, and an image V0
P13 and VOP14 indicate images of the upper layer. The lower layer image VOP11 is an I or P picture, an image VO
P12 is a P picture. Also, the image VO of the upper layer
P13 is a P or B picture, and image VOP14 is a B picture. The enlarged images VM5 and VM6 are images obtained by resolution-converting (enlarging) the image of the lower layer. In the case of FIG. 6, as the image VOP14 of the upper layer, the image VOP13 is used as a reference image for forward prediction, and the enlarged image VM6 is used as a reference image for backward prediction.

In the B picture of the upper layer in the embodiment of the present invention, an image generated from these two predicted reference images is also used as a reference image. For example, in the case of FIG. 6, the image VOP13 of the upper layer and the enlarged image VM6
The generated predicted image PV is used as a reference image of the B picture (VOP14) of the upper layer. That is, when encoding the image VOP14 of the upper layer, the image VOP13, the enlarged image VM6, and the predicted image PV
The images are used as reference images. As described above, the predicted image PV is used as a reference image in a new prediction mode according to the present embodiment in addition to the conventional MPEG method.

More specifically, as described above, when the change (difference) between frames is large as a property of each reference image in the MPEG, the efficiency of motion compensation is reduced and the image at the same time is referred to. In the present embodiment, the enlarged image VM6 generated from the image of the lower layer at the same time is used as the image VOP14 of the upper layer in the region where the difference between the frames is large, since the image has the property that the image is more efficient. Is used for the reference image. Also, the image signal of the upper layer has better image quality than the image signal of the lower layer, and thus, for example, when the change (difference) between frames is small, the prediction error is reduced by using the image of the upper layer. In the present embodiment, in such an area where the difference between frames is small, the image (VOP13) immediately before the same layer (upper layer) is used as a reference image of the image VOP14.

Further, as described above, when the information of the spatial scalable coding of the MPEG system overlaps, that is, when a certain macroblock of the upper layer is coded and the information of the corresponding macroblock of the lower layer (D
In the case of transmitting the (CT coefficient), when the prediction is performed with reference to the image immediately before the same upper layer, the information is transmitted independently in the lower layer and the upper layer, and the information transmitted in the lower layer is transmitted. In the present embodiment, when the DCT coefficients of the corresponding lower-layer macroblock are transmitted, the lower-layer image is used as a prediction reference image because the DCT coefficient of the corresponding lower-layer macroblock is transmitted. In this case, duplication of data transmission is reduced, and coding efficiency is improved.

Further, in a conventional image signal encoding system such as MPEG2, the prediction mode is switched in units of macroblocks, so that in the case of a uniform image area, it is sufficient to switch the prediction mode in units of macroblocks. However, for example, an image that moves complicatedly,
In addition, when the size of the image is small, a region that changes with time and a region that does not change temporally are mixed in the macroblock, which causes a decrease in efficiency. Can be switched on a pixel-by-pixel basis to improve coding efficiency.

[0116] Such prediction mode switching in pixel units can be realized by generating a new reference image in addition to the reference image of the MPEG2 system. The new reference image is generated by switching and copying the reference images of the lower layer and the upper layer in pixel units.

For example, in FIG. 6, the predicted image PV is a reference image for realizing the prediction mode switching on a pixel-by-pixel basis. That is, the predicted image PV is a pixel-by-pixel image VOP.
13 or an image generated by copying from the enlarged image VM6. Therefore, when the predicted image PV is used as the reference image, it is substantially equivalent to switching the prediction mode in pixel units.

The procedure for generating the predicted image PV will be described below with reference to FIG.

When generating a predicted image PV for the image VOP14 of the upper layer, the image VOP
It is determined which pixel of the image 13 or the enlarged image VM6 is used. In the simplest case, the difference between the pixels at the corresponding positions of the lower layer images VOP12 and VOP11 is calculated in pixel units. If the difference value is larger than a certain threshold SL, the lower layer image, that is, the enlarged image VM6 is used. Pixels may be copied, and conversely, if the difference value is equal to or smaller than the threshold value SL, pixels may be copied from the image of the upper layer, that is, the image VOP13. However, this discrimination method cannot suppress the duplication of information of the lower layer and the upper layer. In addition, when an object in the image moves in parallel, since the motion compensation is accurate, the image of the upper layer should be used as the prediction reference image. However, in this method, the difference always increases, and the prediction is performed from the lower layer. Would. Therefore, simply examining the change in the pixel value at the same position cannot cope with the parallel movement. Further, since there are many image sequences including such a translation, it is necessary to cope with the translation.

Therefore, in this embodiment, the value of each pixel after IDCT (that is, the value before performing motion compensation) is checked.
If the absolute value is larger than a certain threshold SL, pixels are copied from the image of the lower layer, that is, the enlarged image VM6. Thus, the prediction image PV may be generated. According to this determination method, when the DCT coefficient is transmitted in the lower layer (in this case, the pixel value after IDCT has a large value), the image of the lower layer is easily selected as a reference image, and the upper layer and the lower layer are more likely to be selected. It is possible to suppress duplication of information.

In order to realize this, first, a difference image DV indicating the inter-frame difference of the image of the lower layer is used.
7 is used. This difference image DV7 is an output of each macroblock after IDCT.

That is, returning to the above-described device configuration, the output signal of the IDCT circuit 29 in FIG. 3 (the locally decoded difference signal of the lower layer) is output to the arithmetic circuit 30 and the output signal of FIG. Is transmitted to the upper layer coding circuit 3 shown in FIG.
Will be recorded. The difference image recorded in the frame memory 60 is the difference image DV7 in FIG. Therefore, the difference image DV7 is a lower layer image VOP.
12 is a differential image when motion compensation is performed using the motion vector used for encoding No. 12. A macroblock that has been intra-coded (intra) in the image VOP12 becomes a decoded image signal of the image VOP12 itself.

In the threshold circuit 62 of FIG. 4, the above-described threshold value SL is set, and when the absolute value of the value (difference value) of each pixel of the difference image DV7 exceeds the threshold value SL. ,
The value of each pixel is output as it is, and when the value of each pixel of the difference image DV7 is equal to or less than the threshold value SL, the value of the pixel is output as 0.

The predicted image PV is generated from the image VOP13 and the enlarged image VM6 by referring to the output signal of the threshold circuit 62. Each pixel of the prediction image PV and the difference image DV7
FIG. 7 shows an example of the correspondence between the respective pixels. FIG. 7 shows a case where the resolution of the upper layer is twice the resolution of the lower layer. In this case, four pixels of the prediction image PV correspond to one pixel of the difference image DV7. That is, the four pixels of the predicted image PV are switched with reference to one pixel in the difference image DV7.

When the pixel value of the corresponding position of the difference image DV7 is larger than the threshold value SL for each pixel of the prediction image PV, the value of the pixel at the same position of the prediction image PV with respect to the pixel of the enlarged image VM6 is calculated. , The pixel value of the predicted image PV.

When the pixel value at the corresponding position of the difference image DV7 is equal to or less than the threshold value SL, that is, when the output of the threshold circuit 62 is 0, the macroblock at the corresponding position of the image VOP12 The pixel of the image VOP13 that has been motion-compensated using the motion vector is used as the pixel value of the predicted image PV.

At this time, the motion vector of the macroblock at the corresponding position in the image VOP12 is converted into a motion vector according to a flag indicating the magnification of the upper layer with respect to the lower layer. For example, when the resolution of the upper layer is twice the resolution of the lower layer, the value of the motion vector is doubled. From the same position as that pixel of the predicted image PV,
Images VO at different positions by the motion vector after the conversion
The pixel value of P13 is set as the pixel value of the predicted image PV.

In order to do this, in FIG. 4, the motion vector supplied from the lower layer is input to the scale conversion circuit 63 and is doubled. In a macro block (intra) having no motion vector, the motion vector is set to 0. The doubled motion vector is supplied to the motion compensation circuit 64. The motion compensation circuit 64 compensates the motion of the image in the frame memory 51 in accordance with the scale-converted lower-layer motion vector and supplies the image to the frame memory 61.

As described above, the difference signal DV7 in FIG.
Is the frame memory 6 in the image signal encoding configuration of FIG.
0 is recorded. The predicted image PV in FIG. 6 is recorded in the frame memory 61 in FIG.

FIG. 8 shows an example of the predicted image PV. As shown in FIG. 8, in a region where the absolute value of the pixel of the difference image DV7 is equal to or less than the threshold value SL, the image of the upper layer is used. Can be

FIG. 9 shows a flow of a method of generating the above-described predicted image PV.

In FIG. 9, first, in step ST1, a difference image is formed from the values after IDCT of the lower layer. In the next step ST2, it is determined whether or not the value of the difference image at the corresponding position is equal to or smaller than the threshold value SL. If the value of the difference image at the corresponding position is equal to or smaller than the threshold value SL in step ST2, step ST5
Proceed to. In this step ST5, the pixel value at the same position in the enlarged image VM6 of the lower layer is copied.

On the other hand, if it is determined in step ST2 that the value is larger than the threshold value SL, the process proceeds to step ST3.
In step ST3, the motion vector of the lower layer is doubled. Then, in the next step ST4, motion compensation is performed using the motion vector, and the predicted image PV is constructed by copying from the reference image of the upper layer.

In the B picture of the upper layer in the present embodiment, as described above, three reference images are used to generate a predicted reference image. This prediction mode is determined for each macroblock.

Here, the prediction mode in the encoding of the upper layer will be described.

The prediction mode is determined for each macroblock, as in the coding of the lower layer. There are the following types of prediction modes for macroblocks in the upper layer.

(1) Intra-frame prediction (intra) macroblock (2) Forward prediction (forward) macroblock (3) Backward prediction (backward) macroblock (4) Bidirectional prediction (bidirectional) macroblock (5) Pixel-Based Prediction Macroblock Among the above-mentioned prediction modes, the prediction modes shown in (1) to (4) are the same as the prediction mode of the macroblock of the lower layer. The pixel-based prediction macroblock in the prediction mode shown in (5) refers to the prediction image PV. By referring to the prediction image PV in this way, it is possible to switch the prediction mode substantially in pixel units.
In the pixel unit prediction macro block, pixels at the same position in the prediction image PV are referred to. That is, the motion vector is assumed to be 0, and the motion vector is not encoded.

In the prediction mode of the above five macroblocks, the mode that minimizes the prediction error is selected. The flag indicating the prediction mode is encoded and transmitted by the variable length encoding circuit.

As described above, in the prediction in the upper layer, an image of a scalable layer different from the image to be coded, for example, an image of a lower layer having a lower resolution can be used as a reference image for generating a predicted reference image. is there. Therefore, the upper layer needs to transmit a flag indicating which layer image is used as a reference image for generating a predicted reference image. Therefore, here, a flag indicating which layer image other than the same layer is used to generate a predicted reference image for each scalable layer (a syntax identifier (ref_
layer_id)) is set, encoded, and transmitted. The flag (ref_layer_i) is set for each image (the VOP).
Based on d), a flag (a syntax identifier (ref_select_c to be described later) indicating from which layer the forward (forward) prediction and the backward (backward) prediction are to be predicted.
ode)) is set, encoded, and transmitted. The flag (ref_select_code) in the P picture is shown in the table of FIG. Also, the flag (ref_select_code) in the B picture is shown in the table of FIG. Details of the syntax will be described later.

The reference images of the upper layer and the lower layer are shown in FIG. 10 and FIG.
May be set freely within the range permitted by the table shown in FIG. In the syntaxes of the tables in FIGS. 10 and 11, there is no explicit distinction between spatial scalability and temporal scalability.

Here, in the case of a P picture, the flag (re
When f_select_code) is “11”, the flag (ref_layer
_id) is used as a reference image for generating a predicted reference image at the same time on the layer indicated by (_id). This is used for spatial scalability and SNR scalability. Other modes are used for temporal scalability.

In the case of a B picture, a flag (ref_select
When (_code) is “00”, an image (VOP) decoded immediately before the same layer (VOP) at the same time as the image (VOP) of the layer indicated by the flag (ref_layer_id) is used as a reference image for generating a predicted reference image. This is used for spatial scalability and SNR scalability. Other modes are used for temporal scalability.

Here, returning to FIG. 4, the configuration of the upper layer coding circuit 3 will be described.

Each image (VOP) of each layer is composed of I, P,
Which type of B picture is to be encoded is determined in advance. The motion vector detection circuit 42 shown in FIG. 4 uses a flag (ref_layer_i) based on a preset picture type.
d), (ref_select_code) is set and the motion compensation circuit 52
And outputs it to the variable length coding circuit 46.

The locally decoded image signal of the lower layer image (VOP) is supplied to the upper layer encoding circuit 3 via the resolution conversion circuit 4 in FIG. 1 and to the frame memory 54 in FIG.

The motion vector detecting circuit 42 sets a predetermined reference frame to a flag (ref_layer_id).
Reference is made from the frame memory 41 or the frame memory 51 based on the and the flag (ref_select_code) to perform motion compensation and detect the motion vector. The motion compensation (inter-frame prediction) according to the present invention has four modes: forward prediction, backward prediction, bidirectional prediction, and pixel unit prediction. The prediction mode of the P picture is only forward prediction,
There are four types of picture prediction modes: forward prediction, backward prediction, bidirectional prediction, and pixel unit prediction. The motion vector detection circuit 42 selects a prediction mode that minimizes the prediction error, and generates the prediction mode at that time.

However, in the pixel unit prediction mode, an image signal recorded in the frame memory 61 is referred to. In this mode, the motion vector is set to 0.

At this time, the prediction error is compared with, for example, the variance of the macroblock to be coded. If the variance of the macroblock is smaller, the prediction is not performed on the macroblock and the intra-frame coding is performed. In this case, the prediction mode is intra-picture coding (intra). The motion vector and the prediction mode are input to the variable length coding circuit 46 and the motion compensation circuit 52.

The motion vector detecting circuit 42 is also supplied with a flag FR indicating how many times the size (resolution) of the upper layer is lower than the lower layer. According to the table shown in FIG. 11, the spatio-temporal scalability of the flag (ref_select_code == "00") is obtained in the case of a B picture (image VOP). The forward (forward) prediction is prediction from an image (VOP) decoded using backward prediction immediately before the same layer. The flag indicating the magnification is 1 (the lower layer and the upper layer have the same resolution), and the flag (ref_se
(lect_code == "00") is a special case of spatial scalability and indicates SNR scalability. In this case, for the forward prediction of the upper layer, the motion vector and the prediction mode used by the VOP at the same time of the lower layer are used as they are. Therefore, in this case, the motion vector detection circuit 42 supplies the motion vector and the prediction mode supplied from the lower layer to the motion compensation circuit 52. In this case, the variable length coding circuit 46 does not code the motion vector.

In a B picture (VOP), the pixel unit prediction mode is used only when the flag (ref_select_code = “00”) is set. That is, in the B picture, the pixel unit prediction mode is used only when the image of the lower layer at the same time as the upper layer is used as the reference image.

The motion vector detection circuit 42 has a flag (lower layer CO
D) is supplied and used in detecting a motion vector.

The motion compensation circuit 52 generates a predicted reference image from the reference images stored in the frame memories 51 and 54 based on a predetermined motion vector.
It is input to the arithmetic circuit 43 as a predicted reference image signal.

The motion vector detecting circuit 43 and the motion compensating circuit 52 use the upper layer VOP size and the upper layer VOP offset, the lower layer VOP size and the lower layer VOP offset. However, illustration is omitted in the drawings to prevent complication.

The arithmetic circuit 43 outputs to the DCT circuit 24 a difference signal between the value of the image signal to be encoded in macroblock units and the value of the predicted reference image signal. In the case of an intra macroblock, the arithmetic circuit 43 outputs the macroblock signal to be encoded to the DCT circuit 44 as it is.

In the DCT circuit 44, DCT (Discrete Cosine Transform) processing is performed, and the data is converted into DCT coefficients. This DC
The T coefficient is input to the quantization circuit 45, and the transmission buffer 4
After being quantized in the quantization step corresponding to the data storage amount (buffer storage amount) of No. 7, the quantized data is input to the variable length coding circuit 46.

The variable length coding circuit 46 includes a quantization circuit 45
The quantization data (in this case, I-picture data) supplied from the quantization circuit 45 is converted into a variable-length code such as a Huffman code in accordance with the supplied quantization step (scale). The encoded data is output to the transmission buffer 47. The variable length coding circuit 46 has:
The flag COD from the lower layer coding circuit 5 is supplied and used when performing variable length coding.

The variable length coding circuit 46 also includes a quantization step (scale) from the quantization circuit 45 and a prediction mode (intra-picture prediction, forward prediction,
A mode indicating which of backward prediction, bidirectional prediction, and pixel-based prediction has been set) and a motion vector are input, and these are also variable-length coded.

The variable length coding circuit 46 also includes a flag (upper layer V) indicating the size of the image (VOP) of the upper layer.
An OP size) and a flag (upper layer VOP offset) indicating a position in absolute coordinates are input, and these are also encoded.

The variable-length encoding circuit 46 also receives a flag FR indicating how many times the resolution of the upper layer is higher than the resolution of the lower layer, and this is also encoded.

The transmission buffer 47 temporarily stores the input coded data, and outputs data corresponding to the stored amount to the quantization circuit 45.

The transmission buffer 47 increases the quantization scale of the quantization circuit 45 by the quantization control signal when the storage amount (remaining amount of storable data) increases to the permissible upper limit value. Decrease the amount of data. Conversely, when the storage amount (the remaining amount of data that can be stored) decreases to the allowable lower limit, the transmission buffer 47
Increases the data amount of the quantized data by reducing the quantization scale of the quantization circuit 45 by the quantization control signal. In this way, overflow or underflow of the transmission buffer 47 is prevented.

The data stored in the transmission buffer 47 is read at a predetermined timing and output to the transmission path.

On the other hand, the quantized data output from the quantization circuit 45 is also input to the inverse quantization circuit 48, where it is inversely quantized according to the quantization step supplied from the quantization circuit 45. . An output signal of the inverse quantization circuit 48 (DCT coefficient obtained by inverse quantization) is IDCT (inverse DCT).
T) The signal is input to the circuit 49, where the output signal (image signal or difference signal) is sent to the arithmetic circuit 50 after the inverse DCT processing.

In the arithmetic circuit 50, the IDCT circuit 49
Is an differential signal, the image signal of the motion compensation circuit 52 and the differential signal of the IDCT circuit 49 are added to restore the image signal. When the output signal of the IDCT circuit 49 is an intra macro block, the image signal from the IDCT circuit 50 is output as it is. This image signal is stored in the frame memory 51.

Returning to FIG. 1, the upper layer bit stream and the lower layer bit stream which are the output bit streams of the upper layer coding circuit 3 and the lower layer coding circuit 5, respectively, are input to the multiplexing circuit 6. The multiplexing circuit 6 multiplexes the bit streams of the lower layer and the upper layer and outputs the multiplexed bit stream.
Then, this bit stream is transmitted to the receiving device via the transmission path 7 or supplied to a recording device (not shown) and recorded on the recording medium 8. At this time, if there is another separated image signal, it is further multiplexed and recorded.

Next, FIG. 12 shows an example of an image signal decoding apparatus according to an embodiment corresponding to the image signal encoding apparatus shown in FIG.

In FIG. 12, the bit stream supplied via the transmission line 86 is received by a receiving device (not shown), or the bit stream recorded on the recording medium 87 is reproduced by a reproducing device (not shown). Then, this bit stream is first input to the demultiplexing circuit 81.
The demultiplexing circuit 81 demultiplexes the bit stream, that is, separates and outputs the bit stream of the upper layer and the bit stream of the lower layer.

The bit stream of the lower layer is supplied to the lower layer decoding circuit 85 as it is. The bit stream of the upper layer is supplied to the upper layer decoding circuit 83 via the delay circuit 82.

In the delay circuit 82, the lower layer decoding circuit 85
Then, after delaying the time required to decode one image (for one VOP), the upper-layer bit stream is supplied to the upper-layer decoding circuit 83.

The specific configuration of the lower layer decoding circuit 85 will be described with reference to FIG.

After the lower layer bit stream is temporarily stored in the reception buffer 91, it is supplied to the variable length decoding circuit 92 as encoded data. The variable length decoding circuit 92
The encoded data supplied from the reception buffer 91 is variable-length decoded, and the motion vector and the prediction mode are output to the motion compensation circuit 97, and the quantization step is output to the inverse quantization circuit 93. The quantized data is output to the inverse quantization circuit 93.

The variable length decoding circuit 92 further includes a flag (lower layer VO) indicating the size of the lower layer image (VOP).
P size) and a flag (lower layer VOP offset) indicating the position in the absolute coordinates are decoded.
7 and output for use by the frame memory 96.
These flags, the motion vector, and the prediction mode are also supplied to the upper layer decoding circuit 83 in FIG. further,
The variable length decoding circuit 92 decodes a flag COD indicating whether or not the block is a skip macro block, and
D is supplied to the motion compensation circuit 97 and the upper layer decoding circuit 83 in FIG.

The inverse quantization circuit 93 includes a variable length decoding circuit 92
The supplied quantized data is inversely quantized according to the quantization step also supplied from the variable length decoding circuit 92, and is output to the output signal IDCT circuit 94. The output signal (DCT coefficient) output from the inverse quantization circuit 93 has ID
After the inverse DCT processing is performed by the CT circuit 94, the output signal (image signal or difference image signal) is supplied to the arithmetic circuit 95.

The output signal (difference image signal) of the IDCT circuit 94 is also supplied to and stored in the decoding circuit 85 of the upper layer in FIG.

When the image signal supplied from the IDCT circuit 94 is I-picture data, the image signal is output as it is from the arithmetic circuit 95, and the differential image signal (P or The data is supplied to and stored in the frame memory 96 for generating a predicted reference image signal of (B picture data). This image signal is output as it is to the outside as a reproduced image.

When the input bit stream is a P or B picture, the motion compensating circuit 97 generates a predicted reference image according to the motion vector and the prediction mode supplied from the variable length decoding circuit 92, and calculates the predicted reference image signal by the operation circuit 95.
Output to The arithmetic circuit 95 adds the difference image signal input from the IDCT circuit 94 and the predicted reference image signal supplied from the motion compensation circuit 97, and outputs the result as an output reproduced image. In the case of a P-picture, the image signal from the arithmetic circuit 95 is also input to and stored in the frame memory 96, and is used as a reference image of an image signal to be decoded next.

Returning to FIG. 12, the upper layer bit stream demultiplexed by the demultiplexing circuit 81 is supplied to the upper layer decoding circuit 83 via the delay circuit. A specific configuration of the upper layer decoding circuit 83 will be described with reference to FIG.

The bit stream of the upper layer is temporarily stored in the reception buffer 101 and then supplied to the variable length decoding circuit 102 as encoded data. Variable length decoding circuit 1
Numeral 02 decodes the coded data supplied from the reception buffer 101 into a variable length, outputs the motion vector and the prediction mode to the motion compensation circuit 107, and outputs the quantization step to the inverse quantization circuit 103. The decoded quantized data is output to the inverse quantization circuit 103.

The variable-length decoding circuit 102 also provides a flag (upper layer V) indicating the size of the image (VOP) of the upper layer.
An OP size) and a flag (upper layer VOP offset) indicating a position in absolute coordinates are decoded and output for use by the motion compensation circuit 107 and the frame memory 106.

The variable length decoding circuit 102 also provides an upper layer image (VOP) with respect to a lower layer image (VOP).
Decoding the flag FR indicating the magnification of the size (resolution) of
The signal is output to the motion compensation circuit 107 and the resolution conversion circuit 84 in FIG.

The resolution conversion circuit 84 in FIG. 12 converts the decoded image signal of the lower layer image (VOP) and key signal by filtering according to the flag FR indicating the magnification, and converts the resolution into the upper layer decoding circuit 8.
3 is supplied to the frame memory 119 of FIG.

The variable length decoding circuit 102 also sets flags (ref_layer_id) and (ref_s
elect_code) and outputs it to the motion compensation circuit 107.

The inverse quantization circuit 103 includes the variable length decoding circuit 1
02 is inversely quantized in accordance with the quantization step also supplied from the variable length decoding circuit 102, and is output to the output signal IDCT circuit 104. The output signal (DCT coefficient) output from the inverse quantization circuit 103 is subjected to inverse DCT processing in the IDCT circuit 104, and the output signal (image signal or difference image signal) is supplied to the arithmetic circuit 105.

When the image signal supplied from the IDCT circuit 104 is an I picture, the image signal is output as it is from the arithmetic circuit 105, and the differential image signal (P or B picture data) inputted later to the arithmetic circuit 105 )
Frame memory 106 for generating a predicted reference image of
And stored. This image signal is output to the outside as a reproduced image as it is.

When the input bit stream is a P or B picture, the motion compensation circuit 107
A flag (ref_la indicating the motion vector, prediction mode, lower layer COD, and reference layer supplied from the
yer_id) and (ref_select_code), a predicted reference image is generated from the image signal from the frame memory 106 and the image signal from the frame memory 119, and the predicted reference image signal is output to the arithmetic circuit 105. At this time, the upper layer VOP size and the upper layer VOP offset, the lower layer VOP size and the lower layer VOP offset are used. In the arithmetic circuit 105, the IDCT circuit 104
The difference image signal input from the motion compensation circuit 107 and the prediction reference image signal supplied from the motion compensation circuit 107 are added and output as an output reproduction image. In the case of a P picture, the arithmetic circuit 10
5 is also input to and stored in the frame memory 106, and is used as a predicted reference image of an image signal to be decoded next.

The upper layer decoding circuit 83 of FIG. 12 generates a prediction reference image in the same manner as the method of generating the prediction image PV of FIG. 6 in the encoding circuit, and stores it in the frame memory 112 of FIG.

The output signal (image signal or difference image signal) of the IDCT circuit 94 in the lower layer decoding circuit 85 of FIG. 13 is supplied to the upper layer decoding circuit 83,
Is supplied to the frame memory 110 at The threshold circuit 111 checks whether the absolute value of the signal recorded in the frame memory 110 exceeds a certain threshold SL, like the threshold circuit 62 in the encoding circuit of FIG. In the following cases, 0 is output, and when it is larger than the threshold value SL, the signal of the frame memory 110 is output as it is.

The predicted reference image signal recorded in the frame memory 112 is transmitted to the frame memory 106 or the frame memory depending on whether the value of the differential image signal at the corresponding position recorded in the frame memory 110 is larger than the threshold value SL. It is generated by copying from the memory 119. If the output of the corresponding position of the threshold circuit 111 is 0, that is, if the difference between frames is small, the image signal of the upper layer, that is, the frame signal 106 is copied. At this time, motion compensation is performed using the motion vector supplied from the lower layer decoding circuit 85 in FIG.

Further, the motion vector supplied from the lower layer decoding circuit 83 is doubled by the scale circuit 114. The doubled motion vector is supplied to the motion compensation circuit 113.
Supplied to In the motion compensation circuit 113, when copying pixels from the frame memory 106, the scale circuit 11
According to the motion vector supplied from No. 4, motion compensation is performed and pixels are copied.

When generating a predicted reference image signal to be recorded in the frame memory 112, if the corresponding output of the threshold circuit 111 is other than 0, the image of the lower layer, that is, the image signal recorded in the frame memory 119 Is copied (the motion compensation is not performed).

The prediction reference image signal generated in this manner is recorded in the frame memory 112 and used for prediction together with the images in the frame memories 106 and 119.

When the flag indicating the magnification is 1 and the flag (ref_select_code == "00"), the motion compensation circuit 107 determines the motion vector and the prediction mode supplied from the same time image (VOP) of the lower layer. A prediction reference image signal is generated using the signal and output to the arithmetic circuit 105.

Next, an example of the syntax of the scalable coding will be described. In the following, regarding the description of the syntax, only a portion related to the present invention will be described, and other portions will be omitted here.

FIG. 15 shows the structure of a bit stream. VS (Video Session Class) is one or more VOs
(Video Object Class) is a set of bit streams.

The syntax of the VS is shown below. This syntax conforms to the so-called C ++.

Syntax No. of bits Mnemonic VideoSession () {video_session_start_code sc + 8 = 32 do * {Videoobject ()} while (nextbits_bytealigned () == video_object_start_code) next_start_code () video_session_end_code sc + 8 = 32} Next, VO ( The syntax of Video Object Class) is shown below.

Syntax No. of bits Mnemonic Videoobject () {video_object_start_code sc + 3 = 27 video_object_id 5 do {VideoObjectLayer ()} while (nextbits_bytealigned () == video_object_layer_start_code) next_start_code ()} VO is the whole image or a part of the image Is a bit stream of the object. VOL (Video Object Layer Class) is a class for scalability.

The syntax of VOL is shown below.

Syntax No.of bits Mnemonic VideoObjectLayer () {video_object_layer_start_code sc + 4 = 28 video_object_layer_id 4 video_object_layer_shape 2 if (video_object_layer_shape == "00") {video_object_layer_width 10 video_object_layer_width_layer_shape_effects_layer () || (video_object_layer_shape_effects == "0011") || (video_object_layer_shape_effects == "0100") || (video_object_layer_shape_effects == "0101") video_object_layer_feather_dist 3 if ((video_Object_layer_shape_effects == "0100") || (video_object_layer = ") {for (i = 0; i <video_object_layer_feather_dist; i ++) feathering_filter (); 8 * 15} video_object_layer_sprite_usage 2 if (video_object_layer_sprite_usage! = SPRITE_NOT_USED) {if (video_object_layer_sprite_usage_ = sprite_usage_sprite_usage == ON-LINE_SPRIvrite_sprite_usage_ = sprite_usage 13 for (i = 0; i <no_of_sprite_points; i ++) {sprite_point [i] _x_coordinate 13 sprite_point [i] _y_coordinate 13} lighting_change_in_sprite 1} v ideo_object_layer_quant_type 1 if (video_object_layer_quant_type) {load_intra_quant_mat 1 if (load_intra_quant_mat intra_quant_mat [64] 8 * 64 load_nonintra_quant_mat 1 if (load_nonintra_quant_mat nonintra_quant_mat [64] 8 * 64} Error_resilient_disable 1 Intra_acdc_pred_disable 1 video_object_layer_fcode_forward 2 video_object_layer_fcode_backward 2 Separate_motion_shape_texture 1 if (video_object_layer_sprire_usage == STATIC_SPRITE) sprite_shape_texture () Scalability 1 if (scalability) {ref_layer_id 4 ref_layer_sampling_direc 1 hor_sampling_factor_n 5 hor_sampling_factor_m 5 vert_sampling_factor_n 5 vert_sampling_factor_m 5 enhancement_type 1} do {VideoObjectPlane () _ start_video (_) _ start_video () Identified by the number shown. For example, (video_object_lay
VOL0 where er_id = 0) is a lower layer, and VOL1 where (video_object_layer_id = 1) is an upper layer, for example. The number of scalable layers may be arbitrary.

The scalability is a 1-bit flag indicating whether the VOL is a lower layer or an upper layer. If (scalability = 1), the VOL is a lower layer, otherwise it is an upper layer.

The identifier (ref_layer_id) is a reference image V when a VOL other than the VOL itself is used as a reference image.
This is a flag indicating the OL number. It is transmitted only to upper layers.

The identifiers (hor_sampling_factor_n), (hor
_sampling_factor_m) indicates how many times the horizontal length of the upper layer is greater than the horizontal length of the lower layer (indicating the magnification of the horizontal resolution). The horizontal dimension of the upper layer relative to the lower layer is given by the following equation.

Hor_sampling_factor_n / hor_sampling_factor_m identifier (ver_sampling_factor_n), (ver_sampling_fa
ctor_m) indicates how many times the vertical length of the upper layer is greater than the vertical length of the lower layer (indicating the magnification of the vertical resolution). The vertical dimension of the upper layer relative to the lower layer is given by the following equation.

Ver_sampling_factor_n / ver_sampling_f
actor_m The syntax of VOP (Video Object Plane Class) is shown below.

Syntax No. of bits Mnemonic VideoObjectPlane () {VOP_start_code sc + 8 = 32 do {modulo_time_base 1} while (modulo_time_base! = "0") VOP_time_increment 10 VOP_prediction_type 2 if ((video_object_layer_sprite_usege! = SPRITE_NOT_USED_V___________________________________ described above SPRITE)) {if (no_of_sprite_points> 0) {encode VOP_points ()} if (lighting_change_in_sprite) {lighting_change_factor_encode ()} if (video_object_layer_sprite_useage == STATIC_SPRITE) {return ()} else if (video_object_layer_sprite_us = 8) }} if (video_object_layer_shape! = "00") {VOP_width 10 VOP_heigth 10 VOP_horizontal_mc_spatial_ref 10 marker_bit 1 VOP_vertical_mc_spatial_ref 10 if (scalability && enhancement_type) background_composition 1} disable_sadct 1 if (VOP_prediction) VOP_predictionquantity = type (video_object_layer_shape_effects == "0011") || (video_object_layer_shape_effects == "0101")) {VOP_const ant_alpha 1 if (VOP_constant_alpha) VOP_constant_alpha_value 8} if (! scalability) {if (! separate_motion_shape_texture) if (error_resilience_disable) combined_motion_shape_texture_coding () else do {do {combined_motion_shape_texture_coding ()} 0000 () ext. nextbits_bytealigned ()! = 000 0000 0000 0000 0000 0000) {next_resync_marker () resync_marker 17 macroblock_number 1-12 quant_scale 5}} while (nextbits_bytealigned ()! = 000 0000 0000 0000 0000 0000) else {if (video_object_layer_shape! = "00" ) {do {first_shape_code 1-3} while (count of macroblocks! = total number of macroblocks)} if (error_resilience_disable) {motion_coding () if (video_object_layer_shape! = "00") shape_coding () texture_coding ()} else do {do {motion_coding ()} while (next_bits ()! = "1010 0000 0000 0000 1") motion_marker 17 if (video_object_layer_shape! = "00") shape_coding () do {texture_coding ()} while (nextbits_bytealigned ()! = "0000 0000 0000 0000 ") if (nextbits_bytealigned ()! = "000 0000 0000 0000 0000 0000) {next_resync_marker () resync_marker 17 macroblock_number 1-12 quant_scale 5}} while (nextbits_bytealigned ()! = 000 0000 0000 0000 0000 0000 0000)}} else {if (background_composition) {load_backward_shape 1 if (load_backward ) {backward_shape_coding () load_foward_shape 1 if (load_foward_shape) foward_shape_coding ()}} ref_select_code 2 if (VOP_prediction_type == "01" || VOP_prediction_type == "10") {forward_temporal_ref 10 if (VOP_prediction_type = marker) backwaed_temporal_ref 10}} combined_motion_shape_texture_coding ()} next_state_code ()} The identifier (VOP_width) and (VOP_height) are the VO
This is a flag (VOP size) indicating the size of P.

The identifier (ref_select_code) is a flag indicating which layer image is to be used as a reference image based on the identifier (ref_layer_id) in forward (forward) and backward (backward) prediction. The details are shown in the table shown in FIG. 10 and the table shown in FIG.

FIG. 16 shows I and P pictures (image VO).
9 shows the syntax of a macroblock in P). (CO
D) is a flag indicating whether or not the data of the macro block exists thereafter. In the case of (COD = 1), it indicates that there is no data of the macro block thereafter (ie, a skip macro block). (COD = 0),
Further, a flag is transmitted. (MCBCP) is a flag indicating the type of the macroblock, and predetermined flags and data are transmitted according to the flag.

FIG. 17 shows the syntax of a macroblock in a B picture (VOP). If the corresponding macroblock of the most recently decoded I or P picture (picture VOP) is a skip macroblock (COD = 1), then that macroblock in the B picture (picture VOP) is also a skip macroblock. Become.

(MODB) is a flag indicating the type of macroblock in the B picture (VOP). FIG. 18 shows the variable length code of (MODB). If (MODB = 0), it indicates that there is no more macroblock data. (MODB =
In the case of 10), (CBPB) is not transmitted, and (MBTYPE) is transmitted. In the case of (MODB = 11), (CBPB) and (MBTYPE) are transmitted. Note that x in the drawing indicates the current macroblock.

(CBPB) is a 6-bit flag indicating whether or not a DCT coefficient exists in each block in the macroblock. If (CBPB) is not transmitted, (CBPB) is interpreted as 0, and no DCT coefficient is transmitted in the macroblock.

[0211] (MBTYPE) is a flag indicating the prediction mode of each macroblock in the B picture. Lower layer
(MBTYPE) is shown in FIG. As shown in FIG. 19, the flag transmitted in the macroblock is determined according to (MBTYPE). Note that x in the drawing indicates the current macroblock.

FIG. 20 shows (MBTYPE) in the upper layer. When (MBTYPE = "1"), the pixel unit prediction mode is set. In this case, no motion vector is transmitted.

The conditions for the skip macroblock in the lower layer are as follows.

(A) P picture (VOP) (1) When COD = “1” In this case, the macro block is treated as a skip macro block. The DCT coefficients are all 0, and the motion vector is also treated as 0.

In the case of a P picture (VOP), the conditions for a skipped macroblock are the same as those of the lower layer.

(B) B picture (VOP) (1) The most recently decoded I or P picture (VOP)
The case where the corresponding macroblock of P) is a skip macroblock (COD = 1).

In this case, a skip macro block is set. The prediction is performed in the same way as for a P picture (VOP), and the motion vector is treated as 0.

(2) In cases other than (1) and MODB == "0"
In this case, the macro block is treated as a skip macro block, and the (MBTYPE) of this macro block is set to Direct (H.2
63)). H.263 is encoded in the same manner as the PB picture. At this time, the motion vector is P
Motion vectors of macroblocks at the same position in a picture (VOP) are used.

The conditions for a skipped macroblock in the upper layer (scalability = 1) are as follows.

(A) P picture (VOP) (1) When COD = “1” In this case, the macro block is treated as a skip macro block. The DCT coefficients are all 0, and the motion vector is also treated as 0.

In the case of a P picture (VOP), the conditions for a skipped macroblock are the same as in the lower layer.

(B) Picture (VOP) In a skip macroblock, its prediction mode and reference image should be set to the most generally efficient mode. Therefore, in the macroblock of the upper layer in the case of spatial scalability, it is efficient to use the pixel unit prediction mode.

In the present embodiment, the upper layer B
The conditions of the skip macroblock of the picture (VOP) are as follows.

(1) When ref_select_code == "00" and MOBD == "0".

In the case of the identifier (ref_select_code = “00”), the most recently decoded I or P picture (VO
Regardless of whether the corresponding macroblock of P) is a skip macroblock (COD value), the subsequent data is always transmitted. At this time, it is encoded next (MODB)
Is 0, the block becomes a skipped macroblock, and no further data is transmitted. At this time, the prediction is in the pixel unit prediction mode, and the motion vector is treated as 0.

(2) ref_select_code! = “00” and most recently decoded I or P picture (VOP)
The corresponding macroblock is a skip macroblock (COD = 1).

In this case, a skip macro block is set. The prediction is performed in the same way as for a P picture (VOP), and the motion vector is treated as 0.

(3) ref_select_code! = "00" and MODB == "0" In this case, the macroblock is treated as a skip macroblock, and the (MBTYPE) of this macroblock is direct (H.263).
And H. H.263 is encoded in the same manner as the PB picture. At this time, the motion vector is a P picture (V) decoded immediately before the VOL indicated by (ref_select_code).
In OP), the motion vector of the macroblock at the same position is used.

[0229]

According to the image signal encoding apparatus and method of the present invention, the lower layer image signal is encoded using the reference image signal to generate first encoded data, and the upper layer image signal is referred to. The second encoded data is generated by encoding using the image signal, the first encoded data is decoded to generate the first reference image signal, and the second encoded data is decoded to generate the second encoded data. Is generated using the third reference image signal generated by adaptively switching between the first reference image signal and the second reference image signal for each pixel, thereby performing prediction. It is possible to improve the efficiency and the coding efficiency.

Further, in the image signal decoding apparatus and method of the present invention, coded data composed of a lower layer image signal and an upper layer image signal coded by using a reference image signal, respectively, is received. The decoded lower layer image signal is decoded using the reference image signal, the encoded upper layer image signal is decoded using the reference image signal, and the decoded lower layer image signal is converted to the first reference image signal. An image signal of a higher layer that is used as a decoded image signal is used as a second reference image signal, and is generated by adaptively switching the first reference image signal and the second reference image signal for each pixel. By decoding using the obtained third reference image signal, it is possible to obtain a good decoded image signal.

Further, the recording medium of the present invention is recorded so as to include encoded data composed of an encoded lower layer image signal and an encoded upper layer image signal. Encoding the lower layer image signal using the reference image signal to generate first encoded data, encoding the upper layer image signal using the reference image signal to generate second encoded data, The first encoded data is decoded to obtain the first
Is generated, the second encoded data is decoded, a second reference image signal is generated, and the first reference image signal and the second reference image signal are adaptively switched for each pixel. By encoding using the third reference image signal generated as described above, it is possible to obtain a good image signal by reproducing and decoding this recording medium.

That is, the present invention relates to an apparatus and a method for encoding an image signal of each layer using a predicted reference image, wherein a signal obtained by encoding the image signal of each layer is decoded to generate a reference image for each layer, Also, a prediction reference image in which each pixel of the reference image is adaptively switched is generated, and an image signal of an upper layer is encoded by using the prediction reference image, so that the prediction efficiency is improved in the spatial scalable encoding method. It is possible to improve the coding efficiency.

As described above, a key signal is input when the input image signal is a separated moving object image, but a key signal is input when the input image signal is a separated background image. No (key) signal is input.

In the case where the input image signal is a separated background image, the encoding and decoding of the key signal are omitted.

Further, according to the present invention, in addition to the case where a certain image signal is separated into a plurality of image signals (background image and moving object image) and encoded for each image, a normal moving image signal May be supplied as an input image signal. In this case, the encoding and decoding of the key signal
Will be omitted.

Various modifications and application examples can be considered without departing from the gist of the present invention. Therefore,
The gist of the present invention is not limited to the embodiments.

[Brief description of the drawings]

FIG. 1 is a block circuit diagram showing a schematic configuration of an image signal encoding device according to an embodiment for realizing the image signal encoding device and method of the present invention.

FIG. 2 is a diagram illustrating an example of arrangement of an image of a lower layer and an image of an upper layer.

FIG. 3 is a block circuit diagram showing a specific configuration of a lower layer encoding circuit of the image signal encoding device.

FIG. 4 is a block circuit diagram illustrating a specific configuration of an upper layer encoding circuit of the image signal encoding device.

FIG. 5 is a diagram used to explain encoding in the case of spatial scalability.

FIG. 6 is a diagram used for describing reference images of P and B pictures in an upper layer.

FIG. 7 is a diagram illustrating an example of a correspondence between each pixel of a prediction reference image and each pixel of a difference image.

FIG. 8 is a diagram illustrating an example of a prediction reference image.

FIG. 9 is a flowchart illustrating a flow of a method of generating a prediction reference image.

FIG. 10 shows a flag (ref_select_cod) in a P picture.
It is a figure showing the table of e).

FIG. 11 shows a flag (ref_select_cod) in a B picture.
It is a figure showing the table of e).

FIG. 12 is a block circuit diagram showing a schematic configuration of an image signal decoding device according to an embodiment for realizing the image signal decoding device and method of the present invention.

FIG. 13 is a block circuit diagram showing a specific configuration of a lower layer decoding circuit of the image signal decoding device.

FIG. 14 is a block circuit diagram showing a specific configuration of an upper layer decoding circuit of the image signal decoding device.

FIG. 15 is a diagram used for describing a hierarchical structure of video syntax.

FIG. 16 is a diagram illustrating the syntax of a macroblock in I and P pictures.

FIG. 17 is a diagram illustrating the syntax of a macroblock in a B picture.

FIG. 18 is a diagram illustrating a MODB variable length code which is a flag indicating a macroblock type in a B picture.

FIG. 19 is a diagram illustrating MBTYPE which is a flag indicating a prediction mode of each macroblock in a B picture of a lower layer.

FIG. 20 is a diagram illustrating an MBTYPE in an upper layer.

FIG. 21 is a block circuit diagram illustrating a schematic configuration example of a conventional MP @ ML encoder of the MPEG system.

FIG. 22 is a block circuit diagram illustrating a schematic configuration example of a conventional decoder of MPEG MP @ ML.

FIG. 23 is a block circuit diagram showing a configuration example of a conventional encoder having spatial scalability.

FIG. 24 is a block circuit showing a configuration example of a conventional decoder of spatial scalability.

[Explanation of symbols]

1 image layering circuit, 2 delay circuit, 3 upper layer coding circuit, 4 resolution conversion circuit, 5 lower layer coding circuit, 6 multiplexing circuit, 81 demultiplexing circuit, 82 delay circuit, 83 upper layer decoding circuit,
94 resolution conversion circuit, 95 lower layer decoding circuit

Claims (56)

[Claims] An image signal encoding apparatus for encoding a lower layer image signal and an upper layer image signal respectively representing a predetermined image signal, wherein the lower layer image signal is encoded using a reference image signal. First encoding means for outputting first encoded data, and second encoding means for encoding the upper layer image signal using a reference image signal and outputting second encoded data. First decoding means for decoding the first encoded data to generate a first reference image signal; and decoding the second encoded data to generate a second reference image signal. And a second decoding unit configured to adaptively switch the first reference image signal and the second reference image signal for each pixel. Encoding using the reference image signal of No. 3 Picture signal encoding apparatus. 2. The method according to claim 1, wherein the second encoding unit includes a reference unit that minimizes a prediction error between the first, second, and third reference image signals and an image signal of an upper layer to be encoded. 2. The image signal encoding apparatus according to claim 1, wherein the image signal is selected and encoded using the selected reference image signal. 3. The third reference image signal is obtained by decoding the first reference image signal and the second reference image signal based on a value obtained by decoding the first encoded data. 2. The image signal encoding device according to claim 1, wherein the image signal encoding device is generated by switching adaptively for each pixel. 4. The third reference image signal is obtained by decoding the first reference image signal and the second reference image signal based on a value obtained by decoding the first encoded data. 3. The image signal encoding apparatus according to claim 2, wherein the image signal encoding apparatus is generated by switching adaptively for each pixel. 5. A value obtained by decoding the first encoded data is a difference value between the lower layer image signal and a prediction reference image signal used for encoding the lower layer image signal. 4. The image signal encoding apparatus according to claim 3, wherein: 6. A value obtained by decoding the first encoded data is a difference value between the lower layer image signal and a prediction reference image signal used for encoding the lower layer image signal. 5. The image signal encoding apparatus according to claim 4, wherein: 7. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third
Selects the pixel of the second reference image signal if the difference value is equal to or less than a predetermined threshold value, and selects the first pixel if the difference value exceeds the predetermined threshold value.
6. The image signal encoding apparatus according to claim 5, wherein the image signal is generated by selecting a pixel of the reference image signal. 8. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third
Selects the pixel of the second reference image signal if the difference value is equal to or less than a predetermined threshold value, and selects the first pixel if the difference value exceeds the predetermined threshold value.
7. The image signal encoding apparatus according to claim 6, wherein the image signal is generated by selecting a pixel of the reference image signal. 9. The image signal encoding apparatus according to claim 1, wherein the resolution of the upper layer image signal is higher than the resolution of the lower layer image signal. 10. The method according to claim 1, wherein the first reference image signal is
10. The image signal encoding apparatus according to claim 9, wherein the image signal obtained by decoding the encoded data of (i) is converted into an image signal in accordance with the resolution of the image signal of the upper layer. . 11. The image signal encoding apparatus according to claim 2, further comprising transmission means for transmitting flag information indicating the selected reference image signal and said first and second encoded data. 12. When encoding the upper and lower layer image signals in block units, and when transmitting block data of a lower layer image signal corresponding to a predetermined block of the upper layer image signal to be encoded. 4. The apparatus according to claim 3, wherein the first reference image signal is selected. 13. A method for coding the upper and lower layer image signals in block units and a method for transmitting block data of a lower layer image signal corresponding to a predetermined block of an upper layer image signal to be coded. 5. The apparatus according to claim 4, wherein the first reference image signal is selected. 14. The first reference image signal is a lower layer image signal at the same time as the upper layer image signal to be coded, and the second reference image signal is an upper layer image to be coded. The image signal encoding apparatus according to claim 3, wherein the image signal is an image signal existing in the same hierarchy as the signal. 15. The first reference image signal is an image signal of a lower layer at the same time as an image signal of an upper layer to be encoded, and the second reference image signal is an image of an upper layer to be encoded. The image signal encoding apparatus according to claim 4, wherein the image signal is an image signal existing in the same hierarchy as the signal. 16. An image signal encoding method for encoding a lower layer image signal and an upper layer image signal respectively representing a predetermined image signal, wherein the lower layer image signal is encoded using a reference image signal. A first encoding step of outputting first encoded data, and a second encoding step of encoding the upper layer image signal using a reference image signal and outputting second encoded data. A first decoding step of decoding the first encoded data to generate a first reference image signal; and decoding the second encoded data to generate a second reference image signal. A second decoding step, wherein the second encoding step includes a step of adaptively switching the first reference image signal and the second reference image signal for each pixel. 3 using the reference image signal Picture signal encoding method characterized by reduction. 17. The method according to claim 17, wherein the second encoding step includes the step of, among the first, second, and third reference image signals, for each block.
17. The image signal encoding method according to claim 16, wherein a reference image signal having a minimum prediction error with respect to an image signal of an upper layer to be encoded is selected, and encoding is performed using the selected reference image signal. Method. 18. The method according to claim 18, wherein the third reference image signal is the first reference image signal.
Is generated by adaptively switching the first reference image signal and the second reference image signal for each pixel based on a value obtained by decoding the encoded data of An image signal encoding method according to claim 16. 19. The method according to claim 19, wherein the third reference image signal is the first reference image signal.
Is generated by adaptively switching the first reference image signal and the second reference image signal for each pixel based on a value obtained by decoding the encoded data of The image signal encoding method according to claim 17. 20. A value obtained by decoding the first encoded data is a difference value between the lower layer image signal and a prediction reference image signal used for encoding the lower layer image signal. 19. The image signal encoding method according to claim 18, wherein: 21. A value obtained by decoding the first encoded data is a difference value between the lower layer image signal and a prediction reference image signal used for encoding the lower layer image signal. 20. The image signal encoding method according to claim 19, wherein: 22. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third reference image signal has a difference value equal to or smaller than a predetermined threshold value. Selecting a pixel of the second reference image signal,
21. The image signal encoding method according to claim 20, wherein if the difference value exceeds the predetermined threshold value, the difference value is generated by selecting a pixel of the first reference image signal. 23. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third reference image signal has a difference value equal to or smaller than a predetermined threshold value. Selecting a pixel of the second reference image signal,
22. The image signal encoding method according to claim 21, wherein if the difference value exceeds the predetermined threshold value, the difference value is generated by selecting a pixel of the first reference image signal. 24. The image signal encoding method according to claim 16, wherein the resolution of the upper layer image signal is higher than the resolution of the lower layer image signal. 25. The first reference image signal, comprising:
25. The image signal encoding method according to claim 24, wherein the resolution of the image signal obtained by decoding the encoded data is converted to an image signal adapted to the resolution of the image signal of the upper layer. . 26. The image signal encoding method according to claim 17, further comprising a transmission step of transmitting flag information indicating the selected reference image signal and first and second encoded data. 27. When the upper and lower layer image signals are encoded in block units, and when the block data of the lower layer image signal corresponding to a predetermined block of the upper layer image signal to be encoded is transmitted. 19. The method according to claim 18, wherein the first reference image signal is selected. 28. When the upper and lower layer image signals are encoded in block units, and when the block data of the lower layer image signal corresponding to a predetermined block of the upper layer image signal to be encoded is transmitted. 20. The image signal encoding method according to claim 19, wherein the first reference image signal is selected. 29. The first reference image signal is a lower layer image signal at the same time as the upper layer image signal to be coded, and the second reference image signal is an upper layer image to be coded. 19. The image signal encoding method according to claim 18, wherein the image signal is an image signal existing in the same hierarchy as the signal. 30. The first reference image signal is an image signal of a lower layer at the same time as an image signal of an upper layer to be encoded, and the second reference image signal is an image of an upper layer to be encoded. 20. The image signal encoding method according to claim 19, wherein the image signal is an image signal existing in the same hierarchy as the signal. 31. An image signal decoding apparatus for receiving encoded data including an encoded lower layer image signal and an encoded upper layer image signal, and decoding the encoded data. The lower-layer image signal and the encoded upper-layer image signal are each encoded using a reference image signal, and receiving means for receiving the encoded data; and the encoded lower-layer image signal By using a reference image signal, and outputting the decoded lower-layer image signal using the reference image signal. And second decoding means for outputting the decoded upper-layer image signal. The lower-layer image signal decoded by the first decoding means is used as a first reference image signal. Second decryption The upper layer image signal decoded in the stage is used as a second reference image signal, and the second decoding means adapts the first reference image signal and the second reference image signal for each pixel. An image signal decoding apparatus for decoding by using a third reference image signal generated by performing the switching. 32. The encoded data includes flag information, and the flag information is used for decoding an image signal of an upper layer among the first, second, and third reference image signals. 32. The image signal decoding apparatus according to claim 31, wherein a reference image signal to be displayed is indicated in block units, and wherein the second decoding means decodes using a reference image signal indicated by the flag information. 33. The method according to claim 31, wherein the third reference image signal is the first reference image signal.
Is generated by adaptively switching the first reference image signal and the second reference image signal for each pixel based on a value obtained by decoding the encoded data of The image signal decoding device according to claim 31. 34. The method according to claim 34, wherein the third reference image signal is the first reference image signal.
Is generated by adaptively switching the first reference image signal and the second reference image signal for each pixel based on a value obtained by decoding the encoded data of 33. The image signal decoding device according to claim 32. 35. A value obtained by decoding the first coded data is a difference value between the lower layer image signal and a prediction reference image signal used for coding the lower layer image signal. The image signal decoding device according to claim 33, wherein: 36. A value obtained by decoding the first encoded data is a difference value between the lower layer image signal and a prediction reference image signal used for encoding the lower layer image signal. 35. The image signal decoding device according to claim 34, wherein: 37. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third reference image signal is provided if the difference value is equal to or less than a predetermined threshold value. Selecting a pixel of the second reference image signal,
36. The image signal decoding apparatus according to claim 35, wherein if the difference value exceeds the predetermined threshold value, the difference value is generated by selecting a pixel of the first reference image signal. 38. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third reference image signal is provided if the difference value is equal to or less than a predetermined threshold value. Selecting a pixel of the second reference image signal,
37. The image signal decoding apparatus according to claim 36, wherein if the difference value exceeds the predetermined threshold value, the difference value is generated by selecting a pixel of the first reference image signal. 39. The image signal decoding apparatus according to claim 31, wherein the resolution of the upper layer image signal is higher than the resolution of the lower layer image signal. 40. The first reference image signal, wherein the resolution of the decoded lower-layer image signal is converted to an image signal adapted to the resolution of the upper-layer image signal. Item 40. The image signal decoding device according to Item 39. 41. When the upper and lower layer image signals are encoded in block units, and when the block data of the lower layer image signal corresponding to a predetermined block of the upper layer image signal to be encoded is transmitted. 33. The image signal decoding apparatus according to claim 32, wherein the first reference signal is selected. 42. The first reference image signal is a lower-layer image signal at the same time as the upper-layer image signal to be coded, and the second reference image signal is an upper-layer image signal to be coded. 33. The image signal decoding device according to claim 32, wherein the image signal is an image signal existing in the same hierarchy as the signal. 43. An image signal decoding method for receiving encoded data including an encoded lower layer image signal and an encoded upper layer image signal, and decoding the encoded data. The lower-layer image signal and the encoded upper-layer image signal are each encoded using a reference image signal, a receiving step of receiving the encoded data, and the encoded lower-layer image signal A first decoding step of decoding the decoded lower-layer image signal using a reference image signal, and decoding the encoded higher-layer image signal using a reference image signal. And a second decoding step of outputting the decoded upper layer image signal. The lower layer image signal decoded in the first decoding step is used as a first reference image signal. The image signal of the upper layer decoded in the second decoding step is used as a second reference image signal, and the second decoding step includes combining the first reference image signal and the second reference image signal. An image signal decoding method, wherein decoding is performed using a third reference image signal generated by adaptively switching each pixel. 44. The encoded data includes flag information, and the flag information is used to decode an image signal of an upper layer among the first, second, and third reference image signals. 44. The image signal decoding method according to claim 43, wherein a reference image signal to be displayed is indicated in block units, and wherein the second decoding step performs decoding using a reference image signal indicated by the flag information. 45. The third reference image signal, wherein:
Is generated by adaptively switching the first reference image signal and the second reference image signal for each pixel based on a value obtained by decoding the encoded data of An image signal decoding method according to claim 43. 46. The third reference image signal, wherein the third reference image signal is
Is generated by adaptively switching the first reference image signal and the second reference image signal for each pixel based on a value obtained by decoding the encoded data of The image signal decoding method according to claim 44. 47. A value obtained by decoding the first encoded data is a difference value between the lower layer image signal and a prediction reference image signal used for encoding the lower layer image signal. 46. The image signal decoding method according to claim 45, wherein: 48. A value obtained by decoding the first encoded data is a difference value between the lower layer image signal and a prediction reference image signal used for encoding the lower layer image signal. 47. The image signal decoding method according to claim 46, wherein: 49. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third reference image signal is provided if the difference value is equal to or smaller than a predetermined threshold value. Selecting a pixel of the second reference image signal,
48. The image signal decoding method according to claim 47, wherein the image signal is generated by selecting a pixel of the first reference image signal if the difference value exceeds the predetermined threshold value. 50. When the resolution of the image signal of the upper layer is higher than the resolution of the image signal of the lower layer, the third reference image signal has a difference value equal to or smaller than a predetermined threshold value. Selecting a pixel of the second reference image signal,
49. The image signal decoding method according to claim 48, wherein if the difference value exceeds the predetermined threshold, the difference value is generated by selecting a pixel of the first reference image signal. 51. The image signal decoding method according to claim 43, wherein the resolution of the upper layer image signal is higher than the resolution of the lower layer image signal. 52. The image processing apparatus according to claim 52, wherein the first reference image signal is an image signal adapted to the resolution of the upper layer image signal by performing resolution conversion on the decoded lower layer image signal. Item 52. The image signal decoding method according to Item 51. 53. When encoding the upper and lower layer image signals in block units, and when transmitting block data of a lower layer image signal corresponding to a predetermined block of the upper layer image signal to be encoded. The method according to claim 44, wherein the first reference signal is selected. 54. The first reference image signal is a lower-layer image signal at the same time as the upper-layer image signal to be encoded, and the second reference image signal is an upper-layer image signal to be encoded. The image signal decoding method according to claim 44, wherein the image signal is an image signal existing in the same hierarchy as the signal. 55. A recording medium that can be decoded by an image signal decoding device, wherein a signal including encoded data including an encoded lower layer image signal and an encoded upper layer image signal is recorded. A first encoding step of encoding the lower layer image signal using a reference image signal and outputting first encoded data; and encoding the upper layer image signal as a reference image. A second encoding step of encoding using a signal to output second encoded data, and a first decoding step of decoding the first encoded data to generate a first reference image signal. , And decodes the second encoded data to produce a second
A second decoding step of generating a reference image signal of the second reference image signal, wherein the second encoding step adaptively switches the first reference image signal and the second reference image signal for each pixel. A recording medium characterized by being encoded by using a third reference image signal generated by the above-mentioned method. 56. The encoded data includes flag information, and the flag information is used to decode an image signal of an upper layer among the first, second, and third reference image signals. 56. The recording medium according to claim 55, wherein a reference image signal to be indicated is indicated in block units.