JPH1013826A - Picture encoder, its method, picture decoder, its method, picture transmitter, picture receiver and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a picture of high quality by multiplexing a part of encoding bit stream of a complex picture to a frame of a small bit quantity at another time. SOLUTION: An encoding difficulty measuring device 701 evaluates the complexity of the picture and calculates an assigning bit quantity bitx necessary for encoding a moving picture corresponding to GOP of MPEG, e.g. with a fixed picture quality at a level more than a prescribed one. Then a controller 703 sorts the assigning bit quantity bitx to a basic picture quality signal assigning bit quantity bit 1 and an improved picture quality signal assigning bit quantity bit 2 based on a total transmittable bit quantity to respectively send to a basic picture quality signal encoding circuit 601 and an improved picture quality signal encoding circuit 602. The circuit 602 encodes and outputs a bit stream bs2 so as to obtain a desired picture quality corresponding to the complexity of the picture, multiplexes and transfers it to another frame small in the basic picture quality signal bit stream bs1.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to an image encoding device and an image encoding method, an image decoding device and an image decoding method, an image transmitting device, an image receiving device, and a recording medium. In particular, for example, a digital moving image signal is recorded on a recording medium such as a magneto-optical disk or a magnetic tape, and the reproduced moving image signal is displayed on a display. An image encoding device and an image encoding method, an image decoding device, and an image decoding method suitable for use when transmitting from a transmission side to a reception side via a transmission path and receiving and displaying the same at the reception side The present invention relates to a method, an image transmission device, an image reception device, and a recording medium.

[0002]

2. Description of the Related Art For example, in a system for transmitting a moving image signal to a remote place such as a video telephone system, the moving image signal is digitized in order to realize high quality transmission. I have. Also, when a moving image signal is recorded on a recording medium, a digitized moving image signal is similarly recorded. Here, since a digitized moving image signal has an enormous amount of data, when recording or transmitting the data, high-efficiency encoding (information compression) of the data is often performed.

[0003] When the data transfer rate of a transmission channel (transmission path) between the moving picture coding apparatus on the transmitting side and the moving picture decoding apparatus on the receiving side is fixed (in the case of a fixed rate). Conventionally, a moving image input in a predetermined unit time is encoded by allocating a fixed bit amount and outputting a bit stream to a transmission path at a fixed bit rate.

FIG. 14 shows a configuration of an example of a conventional moving picture coding apparatus (picture coding apparatus). In this image coding apparatus, an image (moving image) is stored in an MPEG (Moving) image.
Picture Experts Group). That is, a digitized moving image signal is input to the terminal 101, and the moving image signal is input to the arithmetic unit 102 and the motion vector detector (ME).
110. In the motion vector detector 110,
A motion vector between frames of the input moving image signal is detected.

Here, the motion vector detector 110 detects a motion vector by, for example, detecting a reference frame and a small block (macro block) of 16 pixels × 16 lines of a frame to be currently encoded. Is performed by pattern matching (block matching). That is, for example, the pixel value of the i-th pixel from the leftmost and the j-th pixel from the top of the current macroblock (encoding target macroblock) is set as the signal A _ij, and is similarly referred to by an arbitrary motion vector. When a pixel value of a pixel forming a 16 × 16 block in a reference frame is F _ij , a motion vector that minimizes the sum of absolute values E _f represented by the following equation is detected.

E _f = Σ | A _ij −F _ij | where Σ in the above equation represents i and j respectively from 1 to 1
It means summation by changing to an integer value in the range of 6.

[0007] The motion vector detected by the motion vector detector 110 is supplied to a motion compensator (MC) 109 and a VLC unit (variable length encoder) 111. The motion compensator 109 includes a frame memory (FM) 109A, reads a locally decoded image signal stored in the frame memory 109A according to the motion vector from the motion vector detector 110, and ,
The predicted image signal is supplied to the computing units 102 and 108. That is, the motion compensator 109 includes the frame memory 10
The 9A read address is set to an address shifted from the address corresponding to the encoding-target macroblock by an amount corresponding to the motion vector, and an image signal is read therefrom and supplied to the arithmetic units 102 and 108.

The arithmetic unit 102 receives the macroblock to be encoded, which is input to the terminal 101, and the motion compensator 109
Is calculated, and the resulting prediction residual signal is supplied to a DCT (Discrete Cosine Transform) unit 103. Note that, for example, at the start of encoding,
When a scene change occurs, the prediction residual signal is large and the prediction efficiency is low. Therefore, in such a case, the macroblock is coded in a frame (intra). That is, the macroblock to be subjected to intra-frame encoding is supplied to the DCT unit 103 as it is through the arithmetic unit 102.

In the DCT unit 103, two-dimensional DCT processing is performed on the output of the arithmetic unit 102, and the resulting DCT coefficient is supplied to the quantizer 104. In the quantizer 104, the DCT coefficient from the DCT unit 103 is quantized according to the quantization step width supplied from the rate controller (RC) 105, and the resulting quantized coefficient is converted into the inverse quantizer 106 and VLC unit 111
Supplied to

[0010] The VLC unit 111 calculates the quantization coefficient
For example, a variable length encoding process such as run length encoding or Huffman encoding is performed and output. VLC unit 11
In step 1, the variable length encoding process is performed on the quantization step width or the motion vector for each macroblock supplied from the rate controller 105 or the motion vector detector 110, and the result is output. VLC unit 111
Are output to the buffer 112. The buffer 112 absorbs the fluctuation of the bit amount (data amount) by temporarily storing the encoded data from the VLC unit 111, and sets a constant bit rate.
The terminal 113 outputs a bit stream bs0.

The encoded data output from the VLC unit 111 is fed back to the rate controller 105, whereby the rate controller 105
The bit amount of the encoded data output from the C unit 111 is monitored. The rate controller 105 increases the quantization step width when the buffer 112 is likely to overflow, and decreases the quantization step width when the buffer 112 is likely to underflow.
To the quantizer 104, which causes the buffer 112
It is designed to prevent overflow and underflow.

The quantization step width output from the rate controller 105 is supplied to an inverse quantizer 106 and a VLC unit 111 in addition to the quantizer 104.

On the other hand, in the inverse quantizer 106, the quantizer 1
04 is inversely quantized with the same quantization step width as the quantization step width supplied from the rate controller 105, that is, the quantization step width used in the quantizer 104. The resulting D
The CT coefficient is supplied to the inverse DCT unit 107. Inverse DCT
In the unit 107, the DCT coefficient from the inverse quantizer 106 is
The inverse DCT processing is performed, and the result is supplied to the arithmetic unit 108.

The arithmetic unit 108 is supplied with the same data as the predicted image supplied to the arithmetic unit 102 from the motion compensator 109, as described above, in addition to the output of the inverse DCT unit 107. The unit 108 locally decodes the original image by adding the signal (prediction residual signal) from the inverse DCT unit 107 and the predicted image from the motion compensator 109 (however, in the image coding apparatus, When the intra coding is performed, the output of the inverse DCT unit 107 is supplied to the frame memory 109A through the arithmetic unit 108). This decoded image is the same as the decoded image obtained on the receiving side.

The decoded image obtained by the arithmetic unit 108 is supplied to and stored in the frame memory 109A, and is thereafter used as a reference image (reference frame) for an image to be inter-coded (inter-frame coded).

FIG. 15 shows an example of the configuration of an image decoding apparatus for decoding encoded data output from the image encoding apparatus shown in FIG. FIG.
, A bit stream bs0 (encoded data) output at a constant bit rate from the image encoding apparatus is input to the buffer 2
02 and temporarily stored. The encoded data stored in the buffer 202 is sent to a VLD (variable length decoding) unit 2
03, where it is subjected to variable length decoding. The quantization coefficient and the quantization step width for each macroblock obtained by the variable length decoding in the VLD unit 203 are supplied to the inverse quantizer 204, and the motion vector is supplied to the motion compensator 207.

The inverse quantizer 204 inversely quantizes the quantized coefficient from the VLD 203 according to the quantization step width from the VLD 203, and obtains the resulting D
The CT coefficient is output to the inverse DCT unit 205. The inverse DCT unit 205 determines that the DCT coefficient from the inverse quantizer 204 is
The signal is subjected to the CT processing and supplied to the arithmetic unit 206.

The output of the motion compensator 207 is supplied to the arithmetic unit 206 in addition to the output of the inverse DCT unit 205. That is, the motion compensator 207 has a frame memory 207A, and reads a predicted image signal from the frame memory 207A according to the motion vector from the VLD unit 203, as in the case of the motion compensator 109 in FIG. It is supplied to the arithmetic unit 206. The arithmetic unit 206 has an inverse DC
By adding the signal (prediction residual signal) from the T unit 205 and the predicted image from the motion compensator 207, the original image is decoded and supplied to the frame memory 207A. If the output is intra-coded, the output is supplied to the frame memory 207A through the arithmetic unit 206).

The decoded image stored in the frame memory 207A is used as a reference image for an image to be subsequently decoded, and is appropriately read and displayed on, for example, a display (not shown).

[0020]

By the way, as described above, when the data transfer rate of a channel between the video encoder on the transmitting side and the video decoder on the receiving side is constant, In the moving picture coding apparatus, since the quantization step width is determined regardless of the content (complexity) (texture) of the input picture, there is a problem that the picture quality (coding picture quality) of the decoded picture cannot be made constant.

That is, when an image in a certain unit time is flat, the generated code amount is small, and a sufficient bit amount can be allocated. Therefore, in this case, the quantization step width becomes small, and the decoded image obtained by the moving picture decoding apparatus has higher quality than the average. However, when an image in a certain unit time is complicated, the generated code amount is large, and the bit amount allocated for encoding is insufficient. Therefore, in this case, the quantization step width becomes large, and the image quality deteriorates.

In order to keep the image quality of the decoded image constant, the amount of bits allocated for encoding is changed in accordance with the content (complexity) of the input image per unit time, and the bit stream of the variable bit rate is changed. Is effective. However, when the data transfer rate of the transmission path (channel) is constant, it is difficult to transmit a variable-rate bit stream that fluctuates beyond the range that can be absorbed in the buffer 112.

Further, even if the data transfer rate of the transmission path is variable, a bit stream having a bit rate exceeding the maximum rate cannot be transmitted. It was difficult to make the image quality uniform.

[0024] The present invention has been made in view of such a situation, and the image quality of a decoded image is set to a predetermined level (or more).
In addition, uniformity can be maintained.

[0025]

According to a first aspect of the present invention, there is provided an image encoding apparatus, comprising: encoding means for encoding an image in a predetermined unit time into first and second encoded data; Multiplexing for multiplexing the first encoded data and the second encoded data at another time so that the bit amount is equal to or less than a transmittable value that is a value that can be transmitted through a predetermined transmission path, and outputs the multiplexed data. And a conversion means.

According to a tenth aspect of the present invention, in the image encoding method, an image in a predetermined unit time is encoded into first and second encoded data, and the first encoded data at a certain time and another time are encoded. Is multiplexed with the second coded data so that the bit amount is equal to or less than a transmittable value that is a value that can be transmitted through a predetermined transmission path.

An image decoding apparatus according to an eleventh aspect of the present invention is a video decoding apparatus, comprising: separating means for separating data obtained by coding an image into first and second coded data; And a decoding means for decoding the first and second coded data at the same time detected by the detection means.

According to a fourteenth aspect of the present invention, in the image decoding method, data obtained by encoding an image is separated into first and second encoded data, and the first and second encoded data at the same time are separated.
Is characterized by detecting and decoding encoded data.

According to a fifteenth aspect of the present invention, in the image transmitting apparatus, the first image data at a certain time and the second image data at another time are represented by a bit amount that can be transmitted through a predetermined transmission path. A multiplexing means is provided for multiplexing and outputting the multiplexed data so as to be less than a certain transmittable value.

An image receiving apparatus according to a sixteenth aspect of the present invention includes a separating unit that receives transmitted data and separates the data into first and second image data, and separates the first and second image data at the same time. And a detecting means for detecting.

According to a seventeenth aspect of the present invention, in the recording medium, an image in a predetermined unit time is encoded into first and second encoded data, and the first encoded data at a certain time is expressed by:
Data obtained by multiplexing the second encoded data at another time such that the bit amount is equal to or less than a transmittable value that is a value transmittable by a predetermined transmission path is recorded. And

In the image encoding apparatus according to the first aspect, the encoding means encodes the image in a predetermined unit time into the first and second encoded data, and the multiplexing means encodes the image at a certain time. The first encoded data and the second encoded data at another time are multiplexed and output such that the bit amount is equal to or less than a transmittable value that is a value that can be transmitted through a predetermined transmission path. Has been made.

[0033] In the image encoding method according to the tenth aspect, the image in a predetermined unit time is divided into the first and second images.
And a second coded data at a certain time and a second coded data at another time, the bit amount of which is a value that can be transmitted by a predetermined transmission path. Multiplexing is performed as follows.

In the image decoding apparatus according to the eleventh aspect, the separating means separates the data obtained by coding the image into first and second coded data, and the detecting means sets the same time. The first and the second encoded data in are detected. The decoding means decodes the first and second encoded data at the same time detected by the detection means.

In the image decoding method of the present invention, data obtained by encoding an image is separated into first and second encoded data, and the first and second encoded data at the same time are separated. The decrypted data is detected and decrypted.

[0036] In the image transmitting apparatus according to the fifteenth aspect, the multiplexing unit converts the first image data at a certain time and the second image data at another time into a predetermined transmission path having a predetermined bit amount. And multiplexes and outputs the multiplexed data so as to be equal to or less than a transmittable value which is a transmittable value.

In the image receiving apparatus according to the present invention, the separating means receives the transmitted data and separates the received data into first and second image data, and the detecting means detects the first and second image data at the same time. And the second image data are detected.

In the recording medium according to the present invention, an image at a predetermined unit time is encoded into first and second encoded data, and the first encoded data at a certain time and the encoded data at another time are encoded. Data obtained by multiplexing the second encoded data so that the bit amount is equal to or less than a transmittable value that is a value that can be transmitted through a predetermined transmission path is recorded.

[0039]

FIG. 1 shows the configuration of an embodiment of an image coding apparatus to which the present invention is applied. In this image encoding apparatus, an input digital moving image signal is encoded into a bit stream having a constant bit rate so that a uniform and high-quality decoded image can be obtained. Output.

That is, a moving image signal to be encoded is supplied to the terminal 1
The moving image signal is supplied to the delay unit 5 and the bit rate controller 7. The bit controller 7 assigns bits to encode a moving image in a predetermined unit time T according to the image content (complexity) of the input moving image signal so that the image quality of the decoded image is constant. The allocated bit amount, which is the amount, is calculated and supplied to the encoding circuit 6.

More specifically, the bit controller 7 includes, for example, an encoding difficulty measuring device 701, a bit rate calculator 702, and a controller 703.
A moving image signal to be encoded is encoded by an encoding difficulty measuring device 701.
Supplied to

The encoding difficulty measuring device 701 (difficulty calculating means) calculates the encoding difficulty of the input image for each unit time T (a quantity representing the complexity of the image). Here, the unit time T can be, for example, a time corresponding to one frame or 15 frames constituting a GOP (Group Of Picture) defined by MPEG. Further, the unit time T can be a time corresponding to a so-called slice or a macroblock. That is, the unit time T may be at least a time at which synchronization with an image to be coded can be achieved. However, the unit time T
Is preferably a time corresponding to an integral multiple of a frame unit.

The encoding difficulty calculated by the encoding difficulty measuring device 701 is supplied to a bit rate calculator 702. The bit rate calculator 702 (assigned bit amount calculating means) is supplied with the encoding difficulty from the encoding difficulty measuring device 701, and also receives a signal from the terminal 2 in the image encoding apparatus for a predetermined time D. The total bit amount B that can be used is supplied, where the total bit amount B and the encoding difficulty are adjusted so that the image quality of the decoded image is constant regardless of the complexity of the input image. Accordingly, an allocated bit amount bitx [t] for each unit time T is calculated. here,
bitx [t] represents an allocated bit amount necessary for encoding a moving image of a unit time T at a time t so that the image quality of the decoded image is a predetermined image quality of a predetermined level or more.

Here, when the transmission path (channel) for transmitting the bit stream output from the terminal 14 has a bit amount of bit 0 that can be transmitted during a predetermined unit time T, the total bit amount B is , Given by:

B = D × (bit0 / T) where D represents the delay time in the delay unit 5 described later, and the larger the delay time D, the more effective the variable rate control, that is, the change in complexity. It is possible to correspond to a large image.

As described above, the method of determining the amount of allocated bits according to the degree of difficulty of encoding is described in, for example, Japanese Patent Application Nos. 6-522981 and 7-31436 previously filed by the present applicant. The number is described in detail.

The allocated bit amount bitx [t] per unit time T obtained by the bit rate calculator 702 is
It is supplied to the controller 703. Controller 703
Is supplied with the total bit amount B from the terminal 2 in the same manner as in the bit rate calculator 702, in addition to the allocated bit amount bitx [t].
tx [t] and the total bit amount B, the unit time T
When a moving image signal for each is encoded into a basic image quality signal or a high quality image signal to be described later, an assigned bit amount bit1 [t] or bit2 [t] to be assigned to each is calculated.

That is, the controller 703 (dividing means)
Basically, if the picture of the image to be encoded is difficult and the allocated bit amount bitx [t] per unit time T is required to be larger than bit0, bit0 is set to the allocated bit amount bit1.
[T], and the insufficient bit amount (bit amount exceeding bit 0) is bit 2 [t]. Further, the picture of the image to be encoded is simple, and the allocated bit amount bitx per unit time T
If [t] is less than or equal to bit 0, the assigned bit amount b
Itx [t] is set to bit1 [t], and bit2 [t] is set to zero (0). As described above, the controller 7
In 03, the allocated bit amount bitx [t] is divided into bit amounts bit1 [t] or bit2 [t] to be allocated to the basic image quality signal or the high image quality improvement signal, respectively.

Here, the allocated bit amounts bit1 [t] and bit2 [t] are converted by the encoding circuit 6
All of the input images supplied to the bit rate controller 7 while the image input to the device is delayed are calculated for each unit time T.

The allocated bit amounts bit1 [t] and bit2 [t] obtained as described above are supplied from the controller 703 to the encoding circuit 6. Note that the assigned bit amounts bit1 [t] and bit2 [t] are
And a controller 12 to be described later via terminals 3 and 4.
Are also being supplied.

On the other hand, in the delay unit 5, the input moving image signal is delayed by a predetermined time D and supplied to the encoding circuit 6. In the encoding circuit 6 (encoding means), the moving image signal delayed by the delay unit 5 is transmitted to a bit rate controller 7.
It is coded according to the allocated bit amount supplied thereto (by allocating the allocated bit amount), whereby the basic image quality signal (first coded data) (first coded data) for maintaining a predetermined basic image quality is obtained. Image data) and a high-quality image signal (second coded data) (second image data) for maintaining the image quality of a complicated image, etc., which is difficult to maintain only with the basic image quality signal. Data) bit stream bs2.

That is, the encoding circuit 602 comprises a basic image quality signal encoding circuit 601 and a high image quality signal encoding circuit 602.
And an input image for each unit time T is converted into an assigned bit amount bit1 [t] supplied from the bit controller 7.
And a bit stream bs1 [t] of a basic image quality signal and a bit stream bs2 [t] of a high quality image signal having a bit amount designated by bit2 [t]. The encoding into the basic image quality signal or the high image quality signal is performed by the basic image quality signal encoding circuit 601 or the high image quality signal encoding circuit 602.
In each case.

The total bit amount of the bit streams bs1 and bs2 output from the encoding circuit 6 per unit time T is variable, and it is difficult to output to a channel (transmission path) having a constant bit rate as it is. Therefore, in the image encoding device of FIG. 1, by processing the bit streams bs1 and bs2 in the subsequent block, a bit stream having a constant bit rate that can be transmitted through the fixed-rate transmission path is output from the terminal 14. The output has been made.

That is, the bit stream b of the basic image quality signal
s1 [t] is supplied from the basic image quality signal encoding circuit 601 to the delay unit 8 (delay means). The delay unit 8 is composed of, for example, a memory (the same applies to the above-described delay unit 5), and temporarily stores the bit stream bs1 to delay the signal by a predetermined time τ, and a multiplexer (MUX) (multiplexing unit) 9 To supply. Here, the longer the delay time τ in the delay unit 8 is, the higher the quality of the decoded image can be obtained when the bit stream finally output from the terminal 14 is decoded.

On the other hand, the bit stream b
s2 [t] is supplied to the memory 11 (delay means),
Temporarily stored. Then, the bit stream bs2 [t] of the high quality image signal stored in the memory 11 is read out at a predetermined timing and supplied to the multiplexer 9. Here, the writing of the bit stream bs2 [t] to the memory 11 and the bit stream bs
The reading of 2 [t] is controlled by the controller 12 in accordance with the assigned bit amounts bit1 [t] and bit2 [t] supplied from the controller 703.

That is, the controller 12 determines that the allocated bit amount bit1 [t-τ] per unit time T with respect to the bit stream bs1 [t-τ] of the basic image quality signal supplied from the delay unit 8 to the multiplexer 9 is bit0. Is not read from the memory 11, the allocated bit amount b
When it1 [t−τ] is less than bit0, the bit stream bs2 of the high quality image signal is read from the memory 11,
The signal is supplied to the multiplexer 9.

In the multiplexer 9, the bit amount of the bit stream bs 1 from the delay unit 8 and the bit stream bs 2 from the memory 11 are less than the value bit 0 (transmittable value) that can be transmitted by the transmission path. (So that the bit rate is less than bit0 / T) and output.

That is, the assigned bit amount per unit time bit1 [t-] for the bit stream bs1 [t-τ] of the basic picture quality signal supplied from the delay unit 8 to the multiplexer 9
When τ] is equal to bit0, the bit stream bs2 is not read from the memory 11, so that the multiplexer 9 sets the bit rate from the delay unit 8 to bit0 /
The T bit stream bs1 is output as it is. When the assigned bit amount bit1 [t−τ] is less than bit0, the bit rate of the bit stream bs1 from the delay unit 8 and the bit stream bs2 from the memory 11 are set to values that can be transmitted by the transmission path. The output is multiplexed so as to be less than bit0 / T.

Therefore, the bit rate of the bit stream bs0 output from the multiplexer 9 does not become higher than the bit rate bit0 / T of the transmission path.

The multiplexer 9 has a TS (Time Stam).
p) The time stamp is supplied from the generator 13 (generation means), and the multiplexer 9 also adds this time stamp to the bit streams bs1 and bs2. Here, the time stamp means a bit stream bs1, b on the decoding side.
This is information (time information) for indicating a time such as a decoding time or a display time of s2.

The bit stream b output from the multiplexer 9
After s0 is supplied to the buffer 10 and temporarily stored,
The signal is output from the terminal 14 at a constant bit rate bit0 / T. Then, this bit stream is supplied to a recording device (not shown) via a transmission line (channel) having a fixed bit rate of bit 0 / T, for example, a recording medium 15 such as an optical disk, a magneto-optical disk, or a magnetic tape.
Will be recorded.

Next, referring to the timing chart of FIG. 2, the operation of the image coding apparatus of FIG. 1 will be further described. FIG. 2A shows image data (input image) to be encoded. Here, in FIG.
b, c, and d indicate images at the unit time T.

FIG. 2B shows a bit rate calculator 702.
, Images a, b, c, and
The time change of the assigned bit amount bitx [t] with respect to each of d is shown. The dotted line in FIG.
The bit amount bit0 that can be transmitted during the unit time T on the transmission path of the fixed rate is shown. In this embodiment, for the images a and c, the bit amount bitx [t] smaller than the bit amount bit0 is: For the images b and d, the bit amount bitx [t] larger than the bit amount bit0
Is assigned.

That is, as described above, the allocated bit amount bi
tx [t] changes according to the content of the input image so that the image quality of the decoded image is constant. Since the pictures a and c have simple patterns, a small bit amount is used. In addition, images b and d are each assigned a large amount of bits because of the difficulty of the picture.

FIG. 2C shows a bit stream bs1 [t] of a basic image quality signal in an image to which a bit amount bitx [t] as shown in FIG.
In the controller 703, the time change of the allocated bit amount bit1 [t] allocated for each unit time T is shown. Note that the dotted line in FIG.
As in the case of, bit0 is represented.

As described above, the allocated bit amount bitx
When [t] (FIG. 2 (B)) is less than bit0, bit
1 [t] is made equal to the allocated bit amount bitx [t], and when the allocated bit amount bitx [t] is equal to or greater than bit0, bit1 [t] is made equal to bit0.

FIG. 2D shows a bit stream bs2 [t] of a high quality signal in an image to which the bit amount bitx [t] is allocated as shown in FIG. 2B.
In the controller 703, the time change of the allocated bit amount bit2 [t] allocated for each unit time T is shown. Note that the dotted line in FIG. As described above, the allocated bit amount bitx [t] is b
When it is equal to or less than it0, bit2 [t] is set to 0. When the allocated bit amount bitx [t] is larger than bit0, bit2 [t] is set to the allocated bit amount bitx.
Insufficient bit amount for [t], that is, bitx
[T] -bit0.

FIG. 2E shows a bit stream bs1.
It represents a temporal change of the bit amount bit1 [t−τ] of the bit stream obtained by delaying [t] by a predetermined time τ. Note that the dotted line in FIG.

The delay unit 8 supplies the multiplexer 9 with a bit stream bs [t-τ] having a bit amount bit1 [t−τ] as shown in FIG.

FIG. 2F shows the bit rate of the bit stream bs0 [t] output from the multiplexer 9. That is, in the multiplexer 9, the bit stream bs2 of the bit amount bit2 [t] shown in FIG.
[T] and the bit amount bit1 [t−
τ] is multiplexed with the bit stream bs1 [t−τ], thereby outputting a bit stream bs0 having a constant bit rate bit0 / T.

FIG. 2G shows the format of the bit stream bs0 [t] output from the multiplexer 9. Note that, in FIG. 3G, a section separated by a solid line corresponds to a unit time. Also, bs1 (x) is
Bit stream bs of basic image quality signal for image x
1 [t], and bs2 (x) indicates a bit stream bs2 [t] of the image quality improvement signal for the image x.

As shown in FIG. 9G, for a simple image a or c, only a basic image quality signal bs1 (a) or bs1 (c) having a bit amount of bit0 or less is generated, while a complicated image a or c is generated. For each of b and d, the basic image quality signal bs1 (b) or bs1 (d)
The high-quality signal bs2 (b) or bs2 (d) is generated in excess of the bit amount bit0. Therefore, the image quality enhancement signal bs2 (b) is multiplexed with the basic image quality signal bs1 (a) having a bit amount less than bit0 and transmitted. The basic image quality signal bs1 having a bit amount equal to bit0
(B) is transmitted as it is. Further, if the bit amount is bi
The image quality enhancement signal bs2 (d) is multiplexed with the basic image quality signal bs1 (c) less than t0, and the bit amount is b.
The basic image quality signal bs1 (d) equal to it0 is transmitted as it is. The dotted line in FIG. 2G indicates that multiplexing has been performed.

As described above, for an image in which the allocated bit amount bitx [t] in the unit time T is more than bit0, the bit stream bs1 of the basic image quality signal is transmitted with the bit amount bit0. bitx
For an image in which [t] is not more than bit0, the bit stream bs1 of the basic image quality signal and the bit stream bs2 of the high quality image signal at another time are multiplexed, and the total bit rate (bit amount) is transmitted. Since the transmission is performed at the bit rate (or lower) of the channel (transmission path), even when a transmission path of a fixed rate is used, the image quality of the decoded image is maintained at a predetermined constant level (or higher). It is possible to maintain uniformity.

Next, FIG. 3 shows an example of the configuration of the coding difficulty level measuring device 701 of FIG. Encoding difficulty measuring instrument 70
Reference numeral 1 denotes a terminal 51, an arithmetic unit 52, a DCT unit 53, a fixed quantizer 54, a fixed inverse quantizer 56, an inverse DCT unit 57, an arithmetic unit 58, a motion compensator 59, a frame memory 59A, and a motion vector detector 60. , VLC unit 61, generated code amount counter 62, and terminal 63.
Has the same configuration as the image encoding device shown in FIG.

That is, the terminal 51, the arithmetic unit 52, the DCT unit 5
3, inverse DCT unit 57, arithmetic unit 58, motion compensator 59, frame memory 59A, motion vector detector 60, VLC
The container 61 or the terminal 63 is the terminal 10 in FIG.
1, arithmetic unit 102, DCT unit 103, inverse DCT unit 10
7, arithmetic unit 108, motion compensator 109, frame memory 109A, motion vector detector 110, VLC unit 11
1 or the terminal 113, respectively.
Therefore, the encoding difficulty measuring device 701 does not include the rate controller 105 and the buffer 112,
14 except that a generated code amount counter 62 is newly provided and a fixed quantizer 54 or a fixed inverse quantizer 56 is provided instead of the quantizer 105 or the inverse quantizer 106, respectively. Is configured in the same manner as the image encoding device.

In the encoding difficulty measuring device 701, each of the fixed quantizer 54 and the fixed inverse quantizer 56 performs quantization or inverse quantization at a predetermined fixed quantization step width. In the generated code amount counter 62, the bit amount generated when the quantization coefficient obtained using such a fixed quantization step width is subjected to variable-length coding is counted every unit time T. It has been done. Then, this count value is output from the terminal 63 as the image encoding difficulty in the unit time T.

Next, FIG. 4 shows the controller 703 of FIG.
3 is a flowchart for explaining the processing of FIG. In the controller 703, first, in step S1, a variable t representing a time is reset to, for example, 0, and the process proceeds to step S2, where the bit stream b
To transmit s2, a variable S that represents, as it were, the surplus bit amount is initialized to an initial value C.

Here, the initial value C is a variable related to the delay time τ in the delay unit 8 in FIG.
_max is given by the following equation.

C _max = τ × (bit 0 / T)

The initial value C is a value equal to or smaller than _Cmax . Initially, the bit stream bs2 of the high-quality image signal can be advanced by the initial value C. Therefore, the larger the value, the more the final value. The bit stream output from the terminal 14 can be of high image quality.

Thereafter, the flow advances to step S3 to determine whether or not the allocated bit amount bitx [t] of the image in the unit time T to be encoded is larger than the bit amount bit0. In step S3, when it is determined that the allocated bit amount bitx [t] is larger than the bit amount bit0, that is, with the bit amount bit0, the image in the unit time T that is the current encoding target is If it is difficult to perform encoding to maintain the image quality at a predetermined constant value, the process proceeds to step S4,
The difference U between x [t] and bit0 is calculated. Here, the difference U is obtained by assigning the bit amount bitx [t] to the bit amount bi.
The bit amount exceeding t0, that is, the bit amount insufficient with the bit amount bit0 alone is shown.

After calculating the difference U, in step S5,
The bit amount bit1 [t] assigned to the basic image quality signal is b
It is set to it0, the process proceeds to step S6, and the difference U is
It is determined whether: If it is determined in step S6 that the difference U is equal to or smaller than the variable S, that is, if there is more bit amount to transmit data of bit amount U exceeding bit amount bit0, step S7 Bit2 to be assigned to the high-quality image signal
[T] is set to U, and the process proceeds to step S9.

In step S6, the difference U is
If it is determined that it is not less than the variable S, that is, the bit amount b
If there is no more bit amount to transmit data having a bit amount U exceeding it0, the process proceeds to step S8, where the assigned bit amount bit2 [t] is set to S, that is,
The remaining bit amount is the allocated bit amount bit2
[T], and the process proceeds to step S9.

In step S9, the bit amount bit2 [t] assigned to the high-quality image signal in step S7 or S8 is subtracted from the variable S.
Is set, and the process proceeds to step S12.

On the other hand, in step S3, when it is determined that the allocated bit amount bitx [t] is not larger than the bit amount bit0, that is, when the allocated bit amount bitx [t] is equal to or less than the bit amount bit0, the image,
If it is possible to perform encoding so as to maintain the image quality of the decoded image at a predetermined constant value, the process proceeds to step S10, where a difference between the bit amount bit0 and the allocated bit amount bitx [t] is calculated. The variable S is incremented.

Then, the process proceeds to a step S11, wherein the bit amount bit1 [t] allocated to the reference image quality signal is bitx
[T], and the bit amount bit2 [t] to be allocated to the image quality improvement signal is set to 0, and the process proceeds to step S12.

In step S12, the variable (time) t is incremented by the unit time T, and the flow advances to step S13 to calculate the bit amounts bit1 [t] and bit2 [t] for all the series of images (frames) to be encoded. Is determined. If it is determined in step S13 that the bit amounts bit1 [t] and bit2 [t] have not been calculated for all the series of images to be encoded, the process returns to step S3, and the image at the next unit time T is determined. The processing of step S3 and subsequent steps is repeated for the target.

In step S13, the bit amount bit1 is set for all the series of images to be encoded.
If it is determined that [t] and bit2 [t] have been calculated, the process ends.

As described above, the controller 703 calculates the bit amounts bit1 [t] and bit2 [t] for all the series of images to be encoded.

Next, FIG. 5 shows a configuration example of the encoding circuit 6 of FIG. In the figure, the same reference numerals are given to portions corresponding to the case in FIG.

As described above, the encoding circuit 6 includes the basic image quality signal encoding circuit 601 and the high image quality signal encoding circuit 602. For example, encoding conforming to the MPEG system (MPEG encoding) Has been made to take place.

The image quality improving signal encoding circuit 602 includes an arithmetic unit 604, a quantizer 605, a VLC unit 606, an inverse quantizer 607, and a rate controller 608 among the blocks constituting the encoding circuit 6 in FIG. , And arithmetic unit 6
09, and the remaining blocks constitute the basic image quality signal encoding circuit 601.

In the basic picture quality signal encoding circuit 601, the rate controller 105 is not the output of the VLC unit 111 but the assigned bit amount bit 1 from the controller 703.
The image is encoded by performing the same processing as in FIG. 14 except that the quantization step width is determined according to [t]. Therefore, in this case, the quantizer 104 performs the quantization with a quantization step width such that the bit amount per unit time T output from the VLC unit 111 is bit1 [t]. I have. In this embodiment, as in the case of FIG. 14, the output of the VLC unit 111 is fed back to the rate controller 105.
The quantization step width is finely adjusted so that the bit amount output from the VLC unit 111 more accurately becomes bit1 [t].

On the other hand, in the image quality improving signal encoding circuit 602, the output of the inverse quantizer 106, that is, the DCT coefficient obtained by inversely quantizing the quantized coefficient output from the quantizer 104 is supplied to the arithmetic unit 604. The DCT coefficient before being quantized, which is output from the DCT unit 103, is also supplied to the arithmetic unit 604. The arithmetic unit 604 calculates D before quantization.
DC obtained by inversely quantizing the quantization coefficient from the CT coefficient
The T coefficient is subtracted, whereby the quantization error in the quantizer 104 is obtained and supplied to the quantizer 605.

The quantizer 605 is connected to the rate controller 6
08, the output of the arithmetic unit 604 is quantized according to the quantization step width supplied. That is, to the rate controller 608, the assigned bit amount bit2 [t] is supplied from the controller 703, where the bit amount output from the VLC unit 606 is the assigned bit amount bit2 [t]. Such a quantization step width is determined and supplied to the quantizer 605, the VLC unit 606, and the inverse quantizer 607. In the quantizer 605, the quantization is performed with such a quantization step width, whereby the unit time T output from the VLC unit 606 is obtained.
The bit amount per unit is set to bit 2 [t].

The quantized coefficient obtained by performing the quantization in the quantizer 605 is supplied to the VLC unit 606 and the inverse quantizer 607. In the VLC unit 606, the quantization coefficient from the quantizer 605 and the quantization step width from the rate controller 608 are subjected to variable-length encoding in the same manner as in the VLC unit 111, and the resulting encoded data (high The bit stream bs2 of the image quality signal is output to the memory 11.

The rate controller 608 has V
The output of the LC unit 606 is fed back. Also in this rate controller 608,
As in the case of the rate controller 105, the quantization step width is finely adjusted based on the feedback amount so that the bit amount output from the VLC unit 606 becomes more accurate bit2 [t]. ing.

On the other hand, in the inverse quantizer 607, the quantizer 6
05 is inversely quantized according to the quantization step width from the rate controller 608, and is supplied to the calculator 609. The output of the inverse quantizer 106 is also supplied to the arithmetic unit 609 in addition to the output of the inverse quantizer 607, where they are added and the inverse DCT unit 1
07. That is, by this, the inverse DCT unit 10
The DCT coefficient 7 is supplied with a corrected DCT coefficient corresponding to the quantization error in the quantizer 104.

The encoding method in the encoding circuit 6 described above is based on the Test Model 5 of MPEG2.
(ISO / IECJTC1 / SC29 / WG11 N0400, April 1993, pp.85)
, Is disclosed as SNR scalability.

Next, FIG. 6 shows a configuration example of the memory 11 of FIG. As shown in the figure, the memory 11 has a ring memory (ring buffer) structure, and when writing the bit stream bs2 [t] of the latest time, if there is no free space, the bit of the oldest time is used. Writing (overwriting) is performed on the area where the stream bs2 is stored.

Next, FIG. 7 is a flowchart for explaining the processing of the multiplexer 9, the memory 11, the controller 12, and the TS generator 13 of FIG. First of all,
In step S21, the controller 12 resets a variable t representing a time to, for example, 0, and proceeds to step S22.
And the allocated bit amount bit1 supplied from the terminal 3
It is determined whether or not [t−τ] is equal to bit0.

In step S22, the allocated bit amount b
If it1 [t−τ] is determined to be equal to bit0, that is, the bit stream bs1 at time t−τ
The allocated bit amount bit1 [t−τ] of [t−τ] is bit
If it is equal to 0, the process proceeds to step S23, in which the controller 12 controls the TS generator 13 so that the time stamp T for the bit stream bs1 [t−τ] is obtained.
S1 is generated and supplied to the multiplexer 9.

Then, in step S 24, the multiplexer 9 converts the bit stream bs 1 [t−τ] into
A time stamp TS1 from the TS generator 13 is added,
Output to the buffer 10. That is, in this case, the multiplexer 9
Then, the output of the delay unit 8 (bit stream bs1) is
The output (bit stream bs2) of the memory 11 is supplied to the buffer 10 without being multiplexed.

On the other hand, when it is determined in step S22 that the allocated bit amount bit1 [t−τ] is not equal to bit0, that is, the allocated bit amount bit1 [t−τ] of the bit stream bs1 [t−τ] at time t−τ. t-τ]
Is smaller than bit0, the controller 12 uses the vacant bit amount to store the bit stream bs of the image quality enhancement signal of the image at a time different from the time t−τ.
2 is transmitted.

That is, the controller 12 determines in step S2
5, the bit stream bs2 stored in the memory 11 is transmitted from the oldest bit stream to the maximum bit stream bs2.
Only the bit amount of 0-bit 1 [t−τ] is read and supplied to the multiplexer 9. Further, the controller 12 sets T
By controlling the S generator 13, a time stamp TS1 for the bit stream bs1 [t−τ] is generated, and a time stamp TS2 for the bit stream bs2 read from the memory 11 (this time stamp TS2 is, for example, A time stamp TS1 added to the bit stream bs1 of the basic image quality signal of the image corresponding to the bit stream bs2 read from the memory 11) is generated and supplied to the multiplexer 9.

The multiplexer 9 determines in step S27
A time stamp T is added to the bit stream bs1 [t−τ].
At the same time as adding S1, a time stamp TS2 is added to the bit stream bs2 read from the memory 11, and these are multiplexed and output to the buffer 10.

After the processing in steps S24 and S27,
In step S28, the variable t is incremented by the unit time T, and the process proceeds to step S29. In step S2, all the series of images (frames) to be encoded are targeted.
It is determined whether the processes from 2 to S28 have been performed.
In step S29, if it is determined that the processing of steps S22 to S28 has not been performed on all the series of images to be encoded, the process returns to step S22, and the process returns to step S22 on the image at the next unit time T. The following processing is repeated.

In step S29, all the series of images to be coded are subjected to steps S22 to S22.
When it is determined that the process of 28 has been performed, the process is terminated.

Next, FIG. 8 shows the configuration of an embodiment of an image decoding device for decoding data encoded by the image encoding device of FIG. At the terminal 20, for example, data recorded on the recording medium 15 (FIG. 1) is reproduced by a reproducing device (not shown), whereby the bit stream bs
0 [t] is input at a constant bit rate bit0 / T. The bit stream bs0 [t] input to the terminal 20 is supplied to the buffer 21,
Temporarily stored. The bit stream bs0 [t] stored in the buffer 21 is read by a separator (DMUX) 22 (separating means) in accordance with the timing of the processing.

The separator 22 outputs the bit stream bs0
Is divided into a bit stream bs1 of the basic image quality signal and a bit stream bs2 of the high quality image signal. Further, the separator 22 separates the time stamp TS1 added thereto from the bit stream bs1, and separates the time stamp TS2 added thereto from the bit stream bs2. Then, the separator 22 stores the bit stream bs1 in the decoding circuit 30 (decoding unit) (processing unit), the bit stream bs2 and the time stamp TS2 in the memory 23 (delay unit),
The time stamp TS1 is supplied to the controller 25.

The decoding circuit 30 includes a basic picture quality signal decoding circuit 3
01, and a high quality signal decoding circuit 302. The bit stream bs 1 from the separator 22 is supplied to the basic quality signal decoding circuit 301.

On the other hand, in the memory 23, the separator 22
Stream bs2 and its time stamp T
S2 is stored. Here, the memory 23 has a ring memory structure similarly to the memory 11 shown in FIG. 6, and when the bit stream bs2 at the latest time is written, when there is no free space, the bit stream at the oldest time is written. The area where bs2 is stored is overwritten. In the case where the bit stream bs2 is overwritten,
Similarly, the time stamp TS2 is overwritten.

The controller 25 (detection means) includes the time stamp TS1 supplied from the separator 22 and the memory 2
3, the bit streams bs1 and bs2 at the same time (constituting the same image) are detected by comparing with the time stamp TS2 stored in the time stamp TS2. That is, the controller 25 sets the time stamp TS1
And TS2 coincide with each other, the timestamp TS2
Is added to the memory 2
5 and supplied to the high-quality signal decoding circuit 302 of the decoding circuit 30.

According to the picture coding apparatus of FIG. 1, basically, the bit stream bs2 is basically transmitted temporally earlier than the corresponding bit stream bs1 (constituting the same picture). Therefore, as described above, it is delayed by being stored in the memory 23, and
Is read out after the bit stream bs1 corresponding to the bit stream bs2 is output.

Then, the basic image quality signal decoding circuit 301 or the high quality image quality signal decoding circuit 302 decodes the bit streams bs1 and bs2 at the same time (constituting the same image), and further adds the decoding results. As a result, the original image is decoded and output from the terminal 26.

Next, the operation of the image decoding apparatus of FIG. 8 will be further described with reference to the timing chart of FIG.

FIG. 9A shows the format of the bit stream bs0 [t] input to the terminal 20 (therefore, the separator 22). Here, a bit stream bs0 [t] similar to that shown in FIG. 2G is input.

FIG. 9B shows the bit rate of the bit stream bs0 [t] in FIG. 9A. That is,
The bit stream bs0 [t] is input at the same constant bit rate bit0 / T as the bit rate output from the image encoding device in FIG.

FIG. 9C shows the bit stream bs1 of the basic picture quality signal separated by the separator 22 from the bit stream bs0 shown in FIG. 9A with the bit amount shown in FIG. 9B. Bit amount bit per unit time T
It represents a time change of 1 [t]. Note that the dotted line in FIG. 9C indicates bit0.

As in the case of FIG. 2C, the bit amount bit1 [t] is determined by the assignment bit amount bitx [t].
When it is equal to or less than it0, it is equal to the assigned bit amount bitx [t]. When the assigned bit amount bitx [t] is equal to or greater than bit0, it is equal to bit0.

The basic picture quality signal decoding circuit 301 includes
A bit stream bs1 having a bit amount bit1 [t] as shown in (C) is supplied.

FIG. 9D shows a bit stream bs2 of a high-quality image signal separated by the separator 22 from the bit stream bs0 shown in FIG. 9A with the bit amount shown in FIG. 9B. Bit amount per unit time T of
This represents a time change of 2 [t]. Note that the dotted line in FIG. 9D represents 0.

In the memory 23 shown in FIG.
The bit stream bs2 having the bit amount bit2 [t] as shown in FIG.

FIG. 9E shows the bit amount bit2 [t] per unit time T of the bit stream bs2 read from the memory 23. Here, the dotted line in FIG. 9E indicates 0 as in the case of FIG. 9D.

From the memory 23, the high picture quality signal decoding circuit 30
2 includes a bit amount bit2 as shown in FIG.
The bit stream bs2 of [t] is supplied.

When looking at FIG. 9 (E) (the same applies to FIG. 9 (D) described above) from left to right, the first convex portion is the image quality enhancement signal bs2 ( b), 2
The second convex portion is an image quality enhancement signal bs2 (d) of the image d.
Respectively.

FIG. 9F shows a temporal change of the bit amount (data amount) for each unit time T input to the decoding circuit 30. Note that the dotted line in FIG.

As shown in FIG. 9F, the decoding circuit 30 stores the bit stream bs1 [t] of the basic image quality signal having the bit amount bit1 [t] shown in FIG.
The data of the bit amount combined with the bit stream bs2 [t] of the image quality improvement signal of the bit amount bit2 [t] shown in (E) is input.

FIG. 9G shows a temporal change of a decoded image obtained by inputting the data of the bit amount shown in FIG.

As shown in FIG. 13G, for each of the simple images a and c, decoding is performed using a basic image quality signal bs1 (a) or bs1 (c) having a bit amount of bit0 or less. For the complex image b or d, respectively, the basic image quality signal bs1 (b) or bs1
(D) and the high-quality signal bs2 (b) or bs2
Decoding is performed using (d).

That is, the bit rate of the bit stream bs0 input to the image decoding apparatus is at most the bit rate bit0 / T of the transmission channel, but the sum of the bit streams bs1 and bs2 input to the decoding circuit 30 is Is a variable value whose maximum value is larger than the bit rate bit0 / T of the transmission channel, and as a result, it is possible to decode a high-quality and uniform moving image.

Next, FIG. 10 is a flowchart for explaining the processing of the memory 23 and the controller 25 of FIG.

In the controller 25, the variable t representing the time is initialized to, for example, 0 in step S31, and the process proceeds to step S32, where the bit stream bs1 [t] supplied from the separator 22 to the basic image quality signal decoding circuit 301.
Is stored in the memory 23.
Is determined to be the same as the time stamp TS2 added to the bit stream bs2 stored most recently.

That is, as described above, the time stamp TS1 added to the bit stream bs1 supplied to the basic picture quality signal decoding circuit 301 is supplied from the separator 22 to the controller 25. Upon receiving the time stamp TS1, the controller 25 reads the time stamp TS2 stored in the memory 23 and added to the oldest bit stream bs2, and reads out the time stamp TS2.
At 32, it is determined whether the time stamps TS1 and TS2 are equal.

In step S32, when it is determined that the time stamps TS1 and TS2 are equal, that is, for the image corresponding to the bit stream bs [t] of the basic image quality signal input to the basic image quality signal decoding circuit 301 now. When the bit stream bs2 of the image quality improvement signal is stored in the memory 23, the process proceeds to step S33, and the controller 25 controls the memory 23 to
The bit stream bs2 (the bit stream bs2 to which the same time stamp TS2 as the time stamp TS1 is added) of the image quality improvement signal is read and supplied to the image quality enhancement signal decoding circuit 302, and the process proceeds to step S34. Therefore, in this case, the image is decoded from the basic image quality signal and the image quality improvement signal.

If it is determined in step S32 that the time stamps TS1 and TS2 are not equal, that is, the time stamps TS1 and TS2 correspond to the bit stream bs1 [t] of the basic image quality signal input to the basic image quality signal decoding circuit 301. Bit stream b of the image quality enhancement signal for the image
When s2 is not stored in the memory 23 (when there is no high-quality signal), the process skips step S33 and proceeds to step S34. Therefore, in this case, the image is decoded only from the basic image quality signal.

In step S34, the variable t is incremented by the unit time T, and the flow returns to step S32 until step S32 is completed until decoding of a series of images is completed.
To S34 are repeated.

Next, FIG. 11 shows a configuration example of the decoding circuit 30 of FIG. In the figure, the same reference numerals are given to portions corresponding to the case in FIG.

In the decoding circuit 30, decoding conforming to the MPEG system is performed.
LD device 203, inverse quantizer 204, inverse DCT device 205,
Arithmetic unit 206, motion compensator 207, frame memory 20
7A, and the arithmetic unit 303 is the basic image quality signal decoding circuit 30.
1, and the arithmetic unit 303, the VLD unit 304, and the inverse quantizer 305 form a high-quality signal decoding circuit 302.

In the high-quality signal decoding circuit 302, the bit stream bs2 of the high-quality signal from the memory 23 is
The variable length decoding is performed in the VLD unit 304, and the resulting quantization coefficient and quantization step width are supplied to the inverse quantization unit 305. In the inverse quantizer 305, VLD
The quantization coefficient from the unit 304 is inversely quantized by the quantization step width from the VLD unit 304, and the resulting DCT coefficient is supplied to the arithmetic unit 303.

On the other hand, in the basic picture quality signal decoding circuit 301,
Bit stream bs of the basic image quality signal from the separator 22
1 is variable-length decoded in the VLD unit 203, and the resulting quantization coefficient and quantization step width are supplied to the inverse quantizer 204, and the motion vector is supplied to the motion compensator 207.

In the inverse quantizer 204, inverse quantization is performed in the same manner as in FIG.
The CT coefficient is supplied to the calculator 303. In the arithmetic unit 303, the DCT coefficients from the inverse quantizers 204 and 305 are added, and the DCT coefficient obtained as a result of the addition is inverse DCT coefficient.
Is supplied to the vessel 205.

Thereafter, the same processing as in the case of FIG. 15 is performed, whereby the decoded image is output from the terminal 26.

Note that the decoding method of the reference image quality signal and the high image quality signal in the decoding circuit 30 described above is also described in FIG.
As in the encoding method of the encoding circuit 6 shown in FIG.
2 is standardized as SNR scalability.

As described above, the bit stream bs0
Is separated into bit streams bs1 and bs2, and the bit streams bs1 and bs2 at the same time are decoded. Therefore, the bit stream bs1 at a certain time and the bit stream bs at another time are decoded.
Thus, it is possible to obtain a decoded image whose image quality is maintained at a predetermined constant level irrespective of its complexity, from a bit stream bs0 obtained by multiplexing 2 with a bit amount of bit0 / T or less.

According to the image decoding apparatus of FIG. 8, the larger the size of the memory 23, the longer the bit stream bs2 of the high quality image signal can be stored.
In this case, the time during which the bit stream bs2 of the high-quality signal stored in the memory 23 can be referred to can be extended, so that the quality of the decoded image can be maintained more reliably. Here, it is desirable that the size of the memory 23 is the same as the memory constituting the delay unit 8 in the image encoding device of FIG.

Further, the image decoding apparatus can be configured without the memory 23 and the high-quality signal decoding circuit 302. In this case, an image may be decoded from the bit stream bs1 of the basic image quality signal. However, in this case, the image quality of a complicated image is deteriorated as in the related art.

From this, the image coding apparatus of FIG.
It can be said that backward compatibility with the conventional image decoding device is easily achieved.

That is, the format of the bit stream bs0 (FIG. 2 (G), FIG. 9 (A)) output from the image coding apparatus of FIG. 1 is different from the conventional one, However, it is usually inconvenient to decode a bit stream different from the above with a conventional image decoding device because it is difficult (because there is no backward compatibility with the conventional image decoding device).

However, the bit stream bs0 output by the image coding apparatus of FIG. 1 is transmitted to a conventional image decoding apparatus by a separating means (for example, a demultiplexer) for separating at least the bit stream bs1 from the bit stream bs0. (Means corresponding to the separator 22 in FIG. 8)
Is provided, only the basic image quality signal can be decoded. Therefore, backward compatibility with the conventional image decoding device can be easily achieved.

Further, according to the image encoding apparatus shown in FIG.
As compared with the conventional case, it is possible to transmit the bit stream bs0 having higher error resistance. That is, the error correction capability of the bit stream bs1 is made stronger than that of the bit stream bs2,
0 is transmitted. In this case, even if the bit stream bs2 is lost during transmission or an uncorrectable error occurs in the bit stream bs2, if the bit stream bs1 can be decoded, the bit stream bs1 is transmitted to the image decoding apparatus. To obtain a decoded image. As described above, according to the image encoding device of FIG. 1, it is possible to improve error tolerance as compared with the conventional case.

In this embodiment, the case where the transmission rate of the transmission path is a constant bit rate bit0 / T has been described. However, the present invention is not limited to this, and is applicable to a transmission path of a variable bit rate. It is.

That is, in this case, for an image to which a bit amount exceeding the maximum bit rate of the transmission path (data transfer rate) is assigned, the bit stream bs1 having the bit amount corresponding to the maximum bit rate is assigned.
And a bit stream bs2 exceeding the bit amount
On the other hand, for an image to which a bit amount equal to or less than the maximum bit rate is assigned, the image is encoded into a bit stream bs1 corresponding to the bit amount.
Then, the bit stream bs1 having the bit amount corresponding to the maximum bit rate is transmitted alone, and the bit stream bs2 is obtained by encoding an image to which a bit amount equal to or less than the bit amount corresponding to the maximum bit rate is assigned. The obtained bit stream bs1 and the multiplexed signal may be transmitted in such a manner that the total bit rate thereof is equal to the maximum bit rate of the transmission path (hereinafter, the same).

Further, in the present embodiment, the bit stream bs2 is basically multiplexed with the bit stream bs1 temporally preceding the bit stream bs1 of the corresponding image and transmitted. The bit stream bs2 can be multiplexed and transmitted to a bit stream bs1 that follows the bit stream bs1 corresponding to the bit stream bs1. However, in this case, in the image decoding device, the bit stream bs2 is changed to the corresponding bit stream bs
1 for storing the bit stream bs1 until the corresponding bit stream bs2 is received (the bit stream bs1).
For example, it is necessary to provide a memory (delay means) (corresponding to the memory 23) between the separator 22 and the reference image quality signal decoding circuit 301 to constitute an image decoding apparatus. is there.

Further, as described in this embodiment, the present invention is applied to a case where motion compensation (MC) and DC
When applied to an image encoding device and an image decoding device using T in combination, it is possible to completely achieve backward compatibility with a conventional image decoding device.

That is, in this case, the bit stream bs1
Has a structure that can be decoded by the conventional image decoding device, and the bit stream bs2 has a structure that does not affect the image decoding process of the conventional image decoding device. May be multiplexed.

More specifically, the bit stream bs1 is
Encoding is performed with the same structure as in the conventional case, and the bit stream bs2 is, for example, MPEG video (IS
User_data (user data) defined in O / IEC117172-2 or 13818-2), or an MPEG system (ISO / IEC11172-).
1 or priva defined in 13818-1)
The encoding may be performed as te_stream (private stream).

FIG. 12 shows the structure of user_data defined in the GOP layer. Note that a GOP is composed of one or more coded images to assist random access, and forms one layer in a coding syntax defined by MPEG. As described above, the time corresponding to the frame of the image forming the GOP can be set as the unit time T.

In the GOP, user_data is defined by a portion indicated by A in FIG. user
_Data_start_code (000 in hexadecimal)
001B2) indicates the start of user_data, and user_data is 000001 in hexadecimal.
Continue until the code is received.

It is to be noted that, since MPEG_user_data is prohibited from including 23 or more bit strings of 0, the bit stream bs2 of the high quality image signal is prohibited.
Needs to have a structure that does not include 23 or more 0 bit strings.

FIG. 13 shows the structure of private_stream defined in the packet layer. private_stream is B in FIG.
As shown by packet_start_code_
prefix (a code of 000001 in hexadecimal),
This is an MPEG packet that starts with the following stream_id (hexadecimal BD code).

The MPEG user_data is permitted to be used by a user for a special application. In a conventional image decoding apparatus that conforms to the MPEG system that cannot recognize the purpose of this user_data, since the user_data is skipped, only the bit stream bs1 is to be decoded.
The bit stream bs1 will be decoded in the same manner as in the related art.

Similarly, private_st of MPEG
The room is also allowed to be used by the user for special applications, and therefore private_st
In the case of using the “ream”, the conventional image decoding device also uses
The private_stream is skipped, and only the bit stream bs1 is to be decoded.

On the other hand, in the image decoding apparatus shown in FIG.
In the separator 22, MPEG user_data or p
By reading the bit stream bs2 set as the “rivate_stream” from the received bit stream bs0, it is possible to decode an image having a predetermined constant level of image quality as described above.

That is, assuming that a video CD (Compact Disc) player is assumed as a conventional image decoding apparatus, for example, a video CD includes about 1.1 MPEG-compliant images.
A bit stream of a moving image encoded at a constant bit rate of 5 Mbps is recorded. As described above, in the case of a video CD, since a moving image must be encoded at a constant bit rate of about 1.15 Mbps, it is difficult to make the quality of a decoded image constant. Meanwhile, video CD
By recording a bit stream with a variable bit rate in the, it is possible to make the quality of the decoded image constant,
In this case, it is difficult for the conventional video CD player to perform the decoding.

Therefore, the image coding apparatus shown in FIG. 1 converts the input moving image into the bit stream bs1 of the basic image quality signal.
And a bit stream bs2 of a basic image quality signal of an image at a certain time and a bit stream bs2 of an image quality enhancement signal of an image at another time, Multiplexing, so that the bit rate can be transmitted, ie, for example,
When the recording medium 15 is a video CD, about 1.1
The bit stream bs0 that outputs a fixed bit rate of 5 Mbps is output.

At this time, in the multiplexer 9, the bit stream bs1 of the basic picture quality signal has a structure that can be reproduced by a conventional picture decoding apparatus, and the bit stream bs2 of the high picture quality signal is the user_dat of the MPEG video.
a or private_str of the MPEG system
eam. Then, the resulting bit stream bs0 is recorded on the recording medium 15, which is a video CD. As a result, the bit stream bs0 is recorded on the video CD in the format as shown in FIG. 2 (G) or FIG. 9 (A).

When such a video CD is reproduced by, for example, a conventional image decoding apparatus (here, a video CD player) as shown in FIG. User_da of MPEG video in the extracted bit stream bs0
ta or private_st of MPEG system
The bit stream bs2 of the high quality signal recorded as the beam is skipped, and only the bit stream bs1 of the basic quality signal is extracted and decoded.

On the other hand, in the image decoding apparatus shown in FIG.
In the separator 22, a bit stream bs1 of the reference image quality signal is separated from the bit stream bs0.
user_data or private_stream
The bit stream bs2 of the high quality signal is separated from m, whereby the respective bit streams bs1 and bs2 are decoded as described above, and an image of a predetermined high level image quality is always reproduced.

As described above, when the bit stream bs1 of the basic image quality signal and the bit stream bs2 of the high quality image signal are multiplexed, the bit stream bs1 of the basic image quality signal can be reproduced by a conventional image decoding apparatus. And a bit stream b of a high quality image signal.
Since s2 has a structure that does not affect the image decoding processing of the conventional image decoding device, the conventional image decoding device can decode a moving image from the bit stream bs1 (backward compatibility). In the image decoding apparatus to which the present invention is applied, a high quality image having a constant image quality utilizing the advantage of a variable bit rate is obtained from the bit stream bs1 of the basic image quality signal and the bit stream bs2 of the high quality image signal. A moving image can be decoded.

The image coding apparatus and the image decoding apparatus to which the present invention is applied as described above can be applied to a recording apparatus or a reproducing apparatus for recording or reproducing an image on the recording medium 15. In addition, the present invention can be applied to, for example, a video phone system, a broadcast device, and other image processing devices that transmit an image to a distant place.

The present invention can be applied to, for example, an apparatus for transmitting and receiving image data at a variable rate.

In this embodiment, the case where the encoding method or the decoding method according to the MPEG system is used has been described. However, the scope of the present invention is the encoding method or the decoding method according to the MPEG system. It is not limited to the conversion method.

[0174]

According to the image encoding apparatus of the first aspect and the image encoding method of the tenth aspect, an image in a predetermined unit time is encoded into first and second encoded data. The first coded data at a certain time and the second coded data at another time are multiplexed such that the bit amount is equal to or less than a transmittable value that is a value that can be transmitted through a predetermined transmission path. Is done. Therefore, an image requiring a bit amount exceeding the transmittable value of the transmission path can be transmitted through the transmission path, and the image quality of the decoded image is maintained at a predetermined constant level (or higher) as a whole. It is possible to do.

According to the image decoding apparatus and the image decoding method of the present invention, the data obtained by encoding the image is separated into the first and second encoded data. Then, the first and second encoded data at the same time are detected and decoded. Therefore, it is possible to decode a high-quality image.

According to the image transmitting apparatus of the present invention, the first image data at a certain time and the second image data at another time have a bit amount that can be transmitted through a predetermined transmission path. Are multiplexed so as to be equal to or less than a transmittable value. Therefore, an image that requires a bit amount exceeding the transmittable value of the transmission path can be transmitted through the transmission path.

According to the image receiving apparatus of the sixteenth aspect, the transmitted data is received, and the first and second data are received.
Image data. Then, the first and second image data at the same time are detected. Therefore, for example, a high-quality image can be decoded.

In the recording medium according to the present invention, an image at a predetermined unit time is encoded into first and second encoded data, and the first encoded data at a certain time and the encoded data at another time are encoded. Data obtained by multiplexing the second encoded data so that the bit amount is equal to or less than a transmittable value that is a value that can be transmitted through a predetermined transmission path is recorded. Therefore, by reproducing this, it is possible to obtain a decoded image whose image quality is maintained at a predetermined constant level (or higher) as a whole.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an embodiment of an image encoding device to which the present invention has been applied.

FIG. 2 is a timing chart for explaining the operation of the image encoding device in FIG. 1;

FIG. 3 is a block diagram showing a configuration example of a coding difficulty level measuring device 701 in FIG. 1;

FIG. 4 is a flowchart for explaining processing of a controller 703 in FIG. 1;

FIG. 5 is a block diagram illustrating a configuration example of an encoding circuit 6 in FIG. 1;

FIG. 6 is a diagram for explaining a configuration of a memory 11 of FIG. 1;

FIG. 7 is a flowchart for explaining processing of a multiplexer 9, a memory 11, a controller 12, and a TS generator 13 of FIG. 1;

FIG. 8 is a block diagram illustrating a configuration of an embodiment of an image decoding device to which the present invention has been applied.

FIG. 9 is a timing chart for explaining the operation of the image decoding device in FIG. 8;

FIG. 10 is a flowchart for explaining processing of a memory 23 and a controller 25 in FIG. 8;

11 is a block diagram illustrating a configuration example of a decoding circuit 30 in FIG.

FIG. 12 is a diagram showing a structure of user_data.

FIG. 13 is a diagram showing a structure of a private_stream.

FIG. 14 is a block diagram illustrating a configuration of an example of a conventional image encoding device.

FIG. 15 is a block diagram illustrating a configuration of an example of a conventional image decoding device.

[Explanation of symbols]

5 delay unit, 6 coding circuit (coding means), 7
Bit rate controller, 8 delay unit (delay unit), 9 multiplexer (multiplex unit), 11 memory (delay unit), 12 controller, 13 TS generator (generation unit), 22 separator (separation unit), 23
Memory (delay means), 25 controller (detection means), 30 decoding circuit (decoding means), 52 operation unit, 53 DCT unit, 54 fixed quantizer, 56
Fixed inverse quantizer, 57 inverse DCT unit, 58 arithmetic unit, 59 motion compensator, 59A frame memory,
60 motion vector detector, 61 VLC device, 6
2 Generated code amount counter, 102 arithmetic unit, 103
DCT unit, 104 quantizer, 106 fixed inverse quantizer, 107 inverse DCT unit, 108 arithmetic unit, 1
09 motion compensator, 109A frame memory, 11
0 motion vector detector, 111 VLC unit, 20
3 VLD device, 204 inverse quantizer, 205 inverse D
CT unit, 206 arithmetic unit, 207 motion compensator, 2
07A frame memory, 301 basic picture quality signal decoding circuit, 302 high picture quality signal decoding circuit, 303 arithmetic unit, 304 VLD unit, 305 inverse quantizer,
601 basic image quality signal encoding circuit, 602 high image quality signal encoding circuit, 604 arithmetic unit, 605 quantizer, 606 VLC unit, 607 inverse quantizer, 6
08 Inverse DCT unit, 609 Rate controller, 7
01 encoding difficulty measuring instrument (difficulty calculating means), 70
2 Bit rate calculator (assigned bit amount calculation means),
703 controller (dividing means)

Claims (18)

[Claims] An encoding unit that encodes an image at a predetermined unit time into first and second encoded data; a first encoded data at a certain time; and a second encoded data at another time. Multiplexing means for multiplexing the encoded data so that the bit amount is equal to or less than a transmittable value which is a value transmittable through a predetermined transmission path, and outputting the multiplexed data. . 2. The image processing apparatus according to claim 1, further comprising: an allocation bit amount calculation unit configured to calculate an allocation bit amount which is a bit amount to be allocated for encoding the image in the predetermined unit time, wherein the encoding unit allocates the image. When the bit amount is equal to or less than the transmittable value, the image is encoded into only the first encoded data. When the allocated bit amount for the image is larger than the transmittable value, the image 2. The method according to claim 1, wherein the first encoded data has an amount equal to the transmittable value, and the second encoded data has a bit amount equal to a difference between the allocated bit amount and the transmittable value. 2. The image encoding device according to 1. 3. The multiplexing means outputs, when an allocated bit amount for the image is equal to or more than the transmittable value, only first encoded data for the image, and outputs an allocated bit for the image. When the amount is smaller than the transmittable value, the first encoded data of the image and the second encoded data of another image are converted so that the bit amount is equal to or less than the transmittable value. 3. The image coding apparatus according to claim 2, wherein the image is multiplexed and output. 4. The apparatus according to claim 1, further comprising: a difficulty calculating unit configured to calculate an encoding difficulty indicating a complexity of the image in the predetermined unit time, wherein the allocated bit amount calculating unit calculates the encoding bit amount by the difficulty calculating unit. The image encoding apparatus according to claim 2, wherein the allocated bit amount is calculated based on encoding difficulty. 5. The apparatus according to claim 1, further comprising a dividing unit configured to divide the allocated bit amount calculated by the allocated bit amount calculating unit into bit amounts to be allocated to the first or second encoded data. The image encoding device according to claim 2. 6. The image encoding apparatus according to claim 1, further comprising a delay unit that delays the first or second encoded data output from the encoding unit. 7. The image encoding apparatus according to claim 1, further comprising a generation unit configured to generate time information about time, wherein the time information is added to the first and second encoded data. apparatus. 8. The encoding means according to claim 1, wherein
2. The image encoding apparatus according to claim 1, wherein encoding is performed by (Moving Picture Experts Group). 9. The multiplexing means according to claim 1, wherein said multiplexing means comprises:
The image encoding apparatus according to claim 8, wherein one of the encoded data is output as user data (user_data) or a private stream (private_stream) defined in MPEG. 10. An image in a predetermined unit time is converted to a first
And a second coded data, wherein the bit amount of the first coded data at a certain time and the second coded data at another time can be transmitted by a predetermined transmission path. The image encoding method is characterized in that multiplexing is performed so as to be equal to or less than a transmittable value. 11. Data obtained by encoding an image is
Separating means for separating into first and second encoded data; detecting means for detecting the first and second encoded data at the same time; and the first and second data at the same time detected by the detecting means An image decoding apparatus comprising: decoding means for decoding the second encoded data. 12. The image decoding apparatus according to claim 11, further comprising delay means for delaying said first or second encoded data separated by said separation means. 13. The first and second coded data for the same image have the same time information about time added thereto, and the decoding unit performs the same processing based on the time information. The image decoding apparatus according to claim 11, wherein the first and second encoded data at a time are decoded. 14. Data obtained by encoding an image is
An image decoding method, comprising separating into first and second encoded data, detecting and decoding the first and second encoded data at the same time. 15. An image transmitting apparatus for transmitting variable-rate first and second image data, wherein the first image data at a certain time and the second image data at another time are An image transmitting apparatus comprising: a multiplexing unit that multiplexes and outputs a bit amount to be equal to or less than a transmittable value that is a value that can be transmitted through a predetermined transmission path. 16. Receiving the transmitted data, and
An image receiving apparatus comprising: a separating unit that separates the first and second image data; and a detecting unit that detects the first and second image data at the same time. 17. An image in a predetermined unit time is defined as a first
And the second coded data at a certain time, and the second coded data at another time, the bit amount of which can be transmitted by a predetermined transmission path. A recording medium characterized by recording data obtained by multiplexing so as to be equal to or less than a transmittable value. 18. The method according to claim 18, wherein the first and second encoded data are obtained by converting the image into a Moving Picture Experts Group (MPEG).
18. Up) encoded data, wherein one of the encoded data is recorded as user data (user_data) or private stream (private_stream) defined in MPEG. A recording medium according to claim 1.