JP2000092485A - Encoding device, decoding device, image processor, and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the image processor which can provides encoded image data matching line capacity used for image transmission and decoding-side performance to the decoding side. SOLUTION: An encoding means 100 is provided with a means (input means) 112, which receives the performance of a decoding means as an output destination and transmission line information or user's desired information, etc. The encoding means 100 selects and encodes a scalable image matching the information in real time and sends it out. The decoding means is provided with a means (output means) which sends out information 112, such as the processing performance of the decoding means and user information inputted by a user interface (specifying means), to the encoding means 100.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an encoding apparatus for compressing image data obtained by digitizing an image signal, a decoding apparatus for expanding the compressed image data, and TECHNICAL FIELD The present invention relates to an image processing apparatus provided with functions of an encoding device and a decoding device, and an image processing system.

[0002]

2. Description of the Related Art Conventionally, JPEG and H.264 have been used as international standards for audio and video coding systems. 261, JPE
G and H. MPEG, which is an improved version of H.261, are known.
At present, called the multimedia era that handles audio and images in an integrated manner, MPEG1, an improved version of MPEG,
Further, MPEG2, which is an improvement of MPEG1, is often used.

[0003] Here, MPEG2 is a moving picture coding standard advanced in response to a demand for higher image quality, and has the following features. (1) Application to not only storage media but also communication and broadcast media is considered. (2) High-definition television (HDTV: High Definition Televisio
n) Extensibility to quality. (3) MPEG1 or H.264. Unlike H.261, encoding that enables not only progressive scanning (non-interlaced) but also interlaced scanning (interlaced) images. (4) Scalability (Scalability)
). (5) The MPEG2 decoder uses the MPEG1 bit
Be able to decode streams. That is, it must have downward compatibility.

[0004] Among these features, the scalability function (4) is one newly introduced in MPEG2, and is roughly classified into three. These are spatial scalability (Spatial Scalability), temporal scalability (Temporal Scalability), and signal to noise ratio (SNR: Singnal to Noise Radio) scalability (SNR Scalability). Hereinafter, an outline of each scalability will be described.

(Spatial scalability) FIG. 13 is a diagram showing an outline of coding of spatial scalability. A layer with a small temporal resolution is called a base layer (Base Layer), and a large layer is a higher layer (Enhancement Layer)
Call. The base layer is to thin out (sub-sample) the original image spatially at a certain ratio,
Instead of lowering the spatial resolution (image quality), this is a layer in which the code amount per frame is reduced, that is, a layer with low image quality and low code amount in spatial resolution. In this base layer, a normal MPE limited between frames (the original high-quality image is not targeted but limited to the images in the base layer)
Encode with G2. On the other hand, a higher layer is a layer having high image quality and high code amount in spatial resolution,
In the high-order layer, the image of the base layer is up-sampled (creating a high-resolution screen by adding pixels such as an average value between pixels of the low-resolution screen), and an image (basic Enlarged image of layer: Expanded Base La
yer), which is predicted and coded not only from the image in the higher layer but also from the upsampled enlarged image. When the high-layer image thus encoded is decoded, the image becomes spatially the same size as the original image, and the image quality depends on the compression ratio. With such spatial scalability, two image sequences can be encoded more efficiently than when two image sequences are individually encoded and transmitted. It is also possible to simultaneously transmit a normal television broadcast and an HDTV broadcast and select an image according to the performance of the receiving side.

(Time Scalability) FIG. 14 is a diagram showing an outline of coding of time scalability. A layer with a small temporal resolution is called a base layer (Base Layer), and a large layer is a higher layer (Enhancement Layer)
Call. Instead of reducing the temporal resolution by reducing the temporal resolution (frame rate) of the original image in units of frames at a certain rate,
A layer in which the amount of codes to be transmitted is reduced, that is, a layer with low image quality and low code amount in terms of time resolution. In this basic layer, encoding is performed using normal MPEG2 limited between frames (the original high-quality image is not targeted, but limited to temporally present, past, and future frames in the basic layer).
On the other hand, a high layer is a layer having high image quality and a high code amount in terms of time resolution. In this high layer, not only the I, P, and B pictures are used in the high layer but also the base layer. Predict and encode using the image inside.
When the image of the higher layer encoded as described above is decoded, the frame rate of the image becomes the same size as the original image, and the image quality depends on the compression ratio. When such time scalability is used, for example, a 30 Hz progressive scan image and a 60 Hz progressive scan image can be transmitted simultaneously and efficiently. Also, a combination of interlace and progressive scanning is possible.
As for time scalability, future MPEG
It is for extension of 2 and is not used at present (it is treated as "Reserved").

(SNR Scalability) FIG.
It is a figure which shows the encoding summary of NR scalability. A layer with low image quality is called a base layer (Base Layer), and a layer with high image quality is called a high-order layer (Enhancement Layer). The base layer refers to a relatively high compression ratio (coarse quantization step, etc.) in the process of coding (compressing) the original image, for example, in the process of “blocking → orthogonal transform → quantization → variable length coding”. (Code size), that is, a layer of low image quality and low code amount in terms of image quality (N / S). In this basic layer, encoding is performed using MPEG1 or MPEG2 (prediction encoding) limited to between frames. On the other hand, the higher layer is a layer having a higher image quality and a higher code amount than the base layer. In the higher layer, an image encoded by the base layer is decoded, and the decoded image is used as an original. Only the error subtracted from the image is encoded in the frame at a relatively low compression ratio (quantization step size smaller than the quantization step size of the base layer). In SNR scalability,
No inter-frame (field) prediction is performed. All are intra frame or field encoded. When such SNR scalability is used, two types of images having different image qualities can be simultaneously encoded and decoded efficiently.

Therefore, as an image coding apparatus adopting the above-mentioned MPEG2, for example, there is an encoder 900 as shown in FIG. As shown in FIG. 16, the encoder 900 includes a conversion circuit 901 to which the RGB data of the image is supplied, a selection circuit 902 to which the output of the conversion circuit 901 is supplied, and a conversion circuit 902 to which the output of the selection circuit 902 is supplied. The first data generation circuit 904 and the second data generation circuit 9
03, the selection circuit 902, the first data generation circuit 90
4 and a block processing circuit 905 to which the output of the second data generation circuit 903 is supplied, and a block processing circuit 9
And an encoding circuit 906 to which an output of the encoding circuit 05 is supplied.

First, the conversion circuit 901 converts 8-bit RGB data into 8-bit 4: 2: 0 YC data.
Convert to bCr data. The selection circuit 902 selects any one of the spatial scalability use mode, the SNR scalability use mode, and the normal mode (scalability non-use mode). The selection in the selection circuit 902 is mainly instructed by a user who uses a decoder described later.

When the use mode of the spatial scalability is selected by the selection circuit 902, the YCbCr data (8-bit data each) obtained by the conversion circuit 901 is passed through the selection circuit 902, whereby the first data generation circuit 904 Supplied to The first data generation circuit 904 generates corresponding base layer and higher layer data from the supplied YCbCr data, and generates a block processing circuit 905.
To supply.

On the other hand, when the use mode of SNR scalability is selected by the selection circuit 902, the conversion circuit 901
The YCbCr data (8-bit data for each) obtained in
The data is supplied to the second data generation circuit 903 via the selection circuit 902. The second data generation circuit 903 includes:
From the supplied YCbCr data, corresponding base layer and higher layer data is generated and supplied to the block processing circuit 905.

When the normal mode is selected by the selection circuit 902, the YCbCr data (each 8-bit data) obtained by the conversion circuit 901 is directly supplied to the block processing circuit 905 via the selection circuit 902. Is done.

The block processing circuit 905 independently converts the supplied YCbCr data into YCbCr data,
The following processing is performed. In other words, a block is configured using a block of n pixels in each of the horizontal and vertical directions as a unit. Further, a, b, and c blocks are collectively formed independently of YCbCr to form a macro block.

The coding circuit 906 performs a predetermined coding process on the data of each macro block obtained by the block processing circuit 905 for each macro block.
For example, after selecting an intra (I) or inter (P or B) predictive coding system and performing a prediction process, an orthogonal transform (DCT) process, a quantization process, a variable length coding (VL)
C) Perform processing. The data on which the encoding process has been performed by the encoding circuit 906 is transmitted or recorded as an MPEG2 bit stream.

As a decoding device corresponding to the encoder 900 described above, for example, there is a decoder 910 as shown in FIG. The decoder 910 basically performs the reverse processing of the encoder 900. As shown in FIG. 17, the decoder 910 is supplied with a header detection circuit 911 to which an MPEG2 bit stream is supplied and an output of the header detection circuit 911. Flag detection circuit 912 and decoding circuit 913
And the signal selection circuit 9 to which the output of the decoding circuit 913 is supplied
14, a first data decoding circuit 915 and a second data decoding circuit 916 to which the output of the signal selection circuit 914 is supplied.
And the signal selection circuit 914 and the first data decoding circuit 91
5 and an image quality selection circuit 917 to which a second data decoding circuit 916 is supplied.

First, the header detection circuit 911 performs the
Decrypt the header information contained in the bit stream of
A control signal corresponding to the header information is generated and supplied to the flag detection circuit 912. The flag detection circuit 912 detects a flag related to scalability from the control signal from the header detection circuit 911, and outputs the flag to the decoding circuit 91.
3, to the signal selection circuit 914 and the image quality selection circuit 917, respectively.

The decoding circuit 913 corresponds to the encoding circuit 906 shown in FIG.
And performs a predetermined decoding process on the MPEG2 bit stream supplied via the flag detection circuit 912 in accordance with the flag.

The signal selection circuit 914 cancels a macro block constituting the data decoded by the decoding circuit 913, and then selects a signal path according to the flag from the flag detection circuit 912. Thereby, the signal selection circuit 1
The data (decoded image data) from which the macro block has been released in 94 is sent to the first data decoding circuit 915 when spatial scalability is used, and to the second data when SNR scalability is used. When the scalability is not used, the scalability is directly supplied to the image quality selection circuit 917.

The first data decoding circuit 915 is provided in accordance with FIG.
6 corresponds to the first data generation circuit 904, and decodes the decoded image data (spatial scalability data) from the signal selection circuit 914 into the original YCbCr data and supplies it to the image quality selection circuit 917.

The second data decoding circuit 916 is provided in the above-mentioned FIG.
6 corresponds to the second data generation circuit 903, and the decoded image data (SNR
The scalability data is decoded into the original YCbCr data and supplied to the image quality selection circuit 917.

The image quality selection circuit 917 includes a flag detection circuit 9
According to the flag from S12, if scalability is used, the first data decoding circuit 915 or the second
Of the YCbCr data from the data decoding circuit 916 to select the image quality corresponding to the base layer, the higher layer, etc.
The image data according to the selection is output.

[0022]

By the way, the encoder 900 and the decoder 9 in MPEG2 as described above.
In the interval between 10, any one of the following three methods (encoding / decoding method) is selected and adopted.

(First Method) Both the encoder and the decoder encode or decode according to a desired data rate. Note that there is only one image quality (resolution).

(Second System) On the encoder side, two types of images (images of the base layer and the higher layer) having different sizes (resolutions) are simultaneously encoded using spatial scalability (see FIG. 13). On the decoder side, an image having a low spatial resolution is restored from the base layer, or the base layer and a higher-level layer are restored in accordance with the performance (data processing capacity and the like) of the own apparatus and a display device connected to the own apparatus. Restore high spatial resolution images from both. Here, for example, the original image is converted to an HDTV image signal (1490 ×
In the case of (a signal composed of 1152 pixels), the spatial scalability indicates the following two of the base layer and the higher layer.
There will be types. Basic layer: A layer of an image of 720 × 576 pixels in which the original image is thinned out in both the horizontal and vertical directions (x and y directions). Higher layer: In addition to forward prediction (P) and bi-prediction (B) of the original image, an image obtained by up-sampling the base layer to the same size as the higher layer is coded for prediction (comparison).

(Third system) On the encoder side, the SNR
Using scalability (see FIG. 15 above), two types of images (images of the base layer and the higher layer) having different code amounts (quantization step sizes) are simultaneously encoded. Accordingly, a low image quality (low bit rate) image is restored from the base layer, or
A high quality (high bit rate) image is restored from both the base layer and the higher layer. Here, in SNR scalability, two types of images having different image qualities can be simultaneously encoded and decoded. That is, it is possible to generate images having the same compression ratio but different compression ratios by using two different quantization step sizes (quantization coefficients) for the same image. At this time, an image with a high compression ratio (an image with a low bit rate) is defined as a base layer,
An error obtained by subtracting the image obtained by restoring the base layer from the original image is defined as a higher layer. Therefore, on the decoder side, a product obtained by adding the base layer and the higher layer becomes a high-quality image with a low bit rate.

Then, of these three methods, (2)
And (3), the encoder 9 shown in FIG.
In the selection circuit 902 of 00, either one of spatial scalability and SNR scalability can be selected.

However, when spatial scalability is selected in the selection circuit 902, the image size of the base layer is uniquely determined by the relationship with the higher layer.
There is no freedom to choose arbitrarily. Similarly, when the SNR scalability is selected, similarly, the frame rate (resolution) of the base layer is uniquely determined by the relationship with the higher layer, and the image size of the base layer can be freely selected. There is no degree.

Therefore, in the conventional encoding device as shown in FIG. 16, when the scalability function is used, it is not possible to select a code amount such as an image size or a frame rate. That is, it is not possible to select a decoding device as an output destination or a factor directly related to a line situation. In short, when an image encoded using any of spatial scalability, SNR scalability, or the like is received on the decoding device side (reception side), the image quality options are: 1. Low-quality image obtained by decoding only the base layer Since the image quality is limited to a high-quality image obtained by decoding both the base layer and the high-order layer, there is a problem that the image quality (decoding speed) cannot be selected according to the performance of the decoding device or the needs of the user.

Therefore, the present invention has been made to eliminate the above-mentioned drawbacks, and is intended to provide an encoding apparatus, a decoding apparatus, an image processing apparatus, and an image processing apparatus capable of providing images of any image quality.
And an image processing system.

[0030]

For such a purpose,
A first invention is an encoding device that generates at least two image data having different resolutions from the same image, encodes the image data according to a predetermined encoding method, and outputs the encoded image data. Input means for inputting, and control means for controlling an encoding operation in accordance with information input by the input means are provided.

According to a second aspect, in the first aspect,
The image data having different resolutions includes image data having different spatial or temporal resolutions.

According to a third aspect, in the first aspect,
The input information includes at least one of the information on the resolution and the capability information on the decoding side as the output destination.

According to a fourth aspect of the present invention, at least two
A decoding device that generates image data having two different resolutions and decodes the image data obtained by encoding these image data according to a predetermined encoding method, wherein at least the specifying means for specifying the resolution; Output means for externally outputting the designation information by the means.

According to a fifth aspect, in the fourth aspect,
The image data having different resolutions includes image data having different spatial or temporal resolutions.

According to a sixth aspect based on the fourth aspect,
The specification means specifies the capability information of the own device.

According to a seventh aspect of the present invention, at least two
An image processing apparatus comprising: an encoding unit that generates image data having two different resolutions, encodes and outputs these image data according to a predetermined encoding method, and a decoding unit that decodes an output of the encoding unit. And the decoding means
It is characterized by including at least a designating means for designating the resolution in the encoding means.

According to an eighth aspect, in the seventh aspect,
The image data having different resolutions includes image data having different spatial or temporal resolutions.

According to a ninth aspect, in the seventh aspect,
The specification means specifies the capability information of the own device.

In a tenth aspect based on the seventh aspect, the decoding means decodes image data having an arbitrary resolution included in the output of the encoding means to obtain a decoded image. A decoding unit that decodes and adds at least two pieces of image data having different resolutions included in the output of the encoding unit to obtain a decoded image of an arbitrary resolution.

According to an eleventh aspect of the present invention, there is provided encoding means for generating at least two image data having different code amounts from the same image, encoding these image data in accordance with a predetermined encoding method, and outputting the encoded image data. And a decoding unit for decoding the output of (1), wherein the decoding unit includes at least a designation unit for designating the code amount in the encoding unit.

In a twelfth aspect based on the eleventh aspect, the designating means includes at least a size and a size of the image data corresponding to infrastructure information including at least one of a decoding capability of the decoding means and a transmission line capacity. It is characterized by including setting means for setting any one of the frame rates.

In a thirteenth aspect based on the eleventh aspect, the decoding means decodes image data having an arbitrary code amount included in the output of the encoding means to obtain a decoded image. And a second decoding unit that decodes and adds at least two pieces of image data having different code amounts included in the output of the encoding unit to obtain a decoded image of an arbitrary resolution. .

According to a fourteenth aspect, encoding means for generating at least two image data having different code amounts from the same image, encoding these image data according to a predetermined encoding method, and outputting the encoded data, A decoding means for decoding the output of the decoding means, further comprising a transmission means for automatically transmitting the information of the decoding means between the encoding means and the decoding means, wherein the encoding means Is characterized in that the image data is encoded and output according to the information sent via the transmission means.

A fifteenth invention is characterized in that, in the fourteenth invention, the encoding means determines a code amount at the time of encoding according to information transmitted via the transmission means. .

In a sixteenth aspect based on the fourteenth aspect, the decoding means selects and decodes image data having an arbitrary code amount from each image data included in the output of the encoding means. And

According to a seventeenth aspect, encoding means for generating at least two image data having different code amounts from the same image, encoding these image data in accordance with a predetermined encoding method, and outputting the encoded image data, A decoding means for decoding the output of the image processing apparatus, further comprising an interactive transmission means for transmitting information of the decoding means between the encoding means and the decoding means.

In an eighteenth aspect based on the seventeenth aspect, the information of the decoding means includes at least a code amount that can be transmitted by the decoding means.

In a nineteenth aspect based on the seventeenth aspect, the decoding means selects and decodes image data having an arbitrary code amount from each image data included in the output of the encoding means. And

The twentieth invention has at least claims 1 to 5.
An image processing system including any one of the encoding device according to any one of claims 3 to 6, the decoding device according to any one of claims 4 to 6, and the image processing device according to any one of claims 7 to 19. It is characterized by.

[0050]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment)

The present invention is implemented by, for example, an encoding device 100 as shown in FIG. This encoding device 100
As shown in FIG. 1, a conversion circuit 101 to which 8-bit RGB data is supplied, a first frame memory 102 to which an output of the conversion circuit 101 is supplied,
Data generation circuit 105 and first blocking processing circuit 107 to which the output of the frame memory 102 is supplied.
And a first encoding circuit 109 to which an output of the first blocking processing circuit 107 is supplied. The first blocking processing circuit 107 includes a first data generation circuit 10
5 is also provided. Further, the encoding apparatus 100 includes a second frame memory 104 to which the output of the conversion circuit 101 is supplied, and a second frame memory 1
A second data generation circuit 106 and a second block processing circuit 108 to which an output of the second block processing circuit 108 is supplied, and a second encoding circuit 110 to which an output of the second block processing circuit 108 is supplied. And the second block processing circuit 10
8, the output of the second data generation circuit 106 is also supplied. Then, the output of the first data generation circuit 105 is also supplied to the second data generation circuit 106, and the output of the first encoding circuit 109 is also supplied to the second encoding circuit 110. ing. Further, the coding apparatus 100 further includes a bit stream generation circuit 111 to which the outputs of the first coding circuit 109 and the second coding circuit 110 are supplied, and a control circuit 103 for controlling the operation of the entire apparatus. It has.

As shown in FIG. 2, the internal configuration of the control circuit 103 includes a CPU 701, a program memory 702 in which various processing programs for executing the operation control of the entire apparatus are stored in a readable manner by the CPU 701, and details thereof will be described later. And an information detection circuit 703 to which external information 112 is supplied. The CPU 701 is supplied with information (hereinafter referred to as “external information”) 112 including external infrastructure information and user request information. Therefore, when the CPU 701 reads and executes various processing programs in the program memory 702, the operation of the encoding device 100 described here is realized.

As shown in FIG. 3, the internal configuration of the first data generation circuit 105 includes a first selector 301 to which the output (YCbCr data) of the first frame memory 102 is supplied, and a first selector 301. The second selector 303 and the sampling circuit 304 to which the output of the second selector 303 is supplied, the frame rate controller 305 to which the output of the second selector 303 is supplied, and the frame rate controller 3
05 and a third selector 302 to which each output of the sampling circuit 304 is supplied. Then, the output of the third selector 302 is supplied to the first blocking processing circuit 107 and the second data generation circuit 106.

As shown in FIG. 4, the internal configuration of the second data generation circuit 106 includes a first selector 401 to which the output (YCbCr data) of the second frame memory 104 is supplied and a first data generation circuit Frame memory 405 to which the output of the circuit 105 (image data of the basic layer) is supplied.
A first difference data generation circuit 403 and a second difference data generation circuit 404 to which respective outputs of the first selector 401 and the frame memory 405 are supplied; a first difference data generation circuit 403 and a second difference Data generation circuit 404
And a second selector 402 to which the respective outputs are supplied. Then, the output of the second selector 402 is supplied to the second blocking processing circuit 106.

In the encoding device 100 as described above,
First, the conversion circuit 101 converts the input image data (each of which is 8-bit RGB data) into a 4: 2: 0 YCbC
r data (each 8-bit data)
The CbCr data is stored in the first frame memory 102 and the second
, Respectively. The first frame memory 102 and the second frame memory 104 each store the YCbCr data from the conversion circuit 101, and the operation control at this time is performed by the control circuit 103 that operates as follows.

That is, in the control circuit 103 (see FIG. 2), the information detection circuit 703
Is interpreted and control information corresponding thereto is supplied to the CPU 701. The CPU 701 uses the control information from the information detection circuit 708 to determine mode information indicating the use or non-use of the scalability function in encoding, the type of the scalability function in the case of the use mode of the scalability function, and various controls for the base layer and the higher layer Information (eg, image size (Statial), frame rate (Temporal), compression ratio (SNR), etc.) of the base layer,
The information (hereinafter, referred to as “encoding control signal”) is transmitted to the first data generation circuit 105 and the second data generation circuit 1.
06 and so on. At the same time, the CPU 70
The information 1 is linked to the functions of the first data generation circuit 105 and the second data generation circuit 106 so that the first frame memory 102 and the second frame memory 104 can read and write data. A reading (R / W: Read / Write) control signal according to the control information from the detection circuit 708 is supplied.

Therefore, the above-mentioned first frame memory 102 and second frame memory 104 store the external information 1
12 operates in accordance with the R / W control signal based on the second data generation circuit 105 and the second data generation circuit 1.
06 and the like also operate according to the encoding control signal based on the external information 112.

The conversion circuit will be described below according to the contents indicated by the external information 112, in particular, the "spatial scalability use mode", "temporal scalability use mode", "SNR scalability use mode", and "scalability function non-use mode". The operation of each circuit after 101 will be described.

(Spatial Scalability Use Mode)

Each of the first frame memory 102 and the second frame memory 104 stores an R / W control signal (external information 11) from the control circuit 103 (specifically, the CPU 701).
In accordance with the control signal for specifying the spatial scalability use mode based on 2), the read / write operation of the YCbCr data from the conversion circuit 101 is performed. As a result, the YCbCr data read from the first frame memory 102 and the second frame memory 104 is transferred to the first data generator 105 and the second data generator 106 via the first data generator 105 and the second data generator 106.
Are supplied to the second blocking processing circuit 107 and the second blocking processing circuit 108.

At this time, the first data generation circuit 105 and the second data generation circuit 106 also receive an encoding control signal from the control circuit 103 (a control signal for specifying a spatial scalability use mode based on the external information 112). The supplied first data generation circuit 105 and second data generation circuit 106 operate according to the encoding control signal.

That is, the first data generation circuit 105
(See FIG. 3 above), the first selector 301
The output destination is switched to the sampling circuit 304 according to the encoding control signal from the control circuit 103, and the YCbCr data from the first frame memory 102 is output. The sampling circuit 304 performs image reduction processing on the YCbCr data from the first selector 301 according to the sub-sample size information included in the encoding control signal from the control circuit 103 to generate image data of the base layer. I do. The image data of the base layer obtained by the sampling circuit 304 is supplied to the third selector 302. The third selector 302 switches the output data to the output of the sampling circuit 304 (basic layer image data) in accordance with the encoding control signal from the control circuit 103. Therefore, the first blocking processing circuit 1
07 is supplied with the image data of the base layer. The image data of the basic layer is also supplied to a second data generation circuit 106 described later.

Thus, the first data generation circuit 1
The image data of the base layer supplied to the first block processing circuit 107 from step 05 is divided into blocks by the first block processing circuit 107, and the first coding circuit 109 performs predetermined block unit coding. The bit stream is supplied to the bit stream generation circuit 111.

On the other hand, in the second data generation circuit 106 (see FIG. 4), the first selector 401 sets the output destination to the first difference data generation circuit 403 according to the encoding control signal from the control circuit 103. By switching, the YCbCr data from the second frame memory 104 is output. At the same time, the frame memory 405 stores the control circuit 103
The first data generation circuit 1
The image data of the basic layer starting from 05 is supplied to the first difference data generation circuit 403. The first difference data generation circuit 403 converts the image data of the base layer from the frame memory 405 into the original image (or the image of the higher layer) in units of frames (or fields) according to the encoding control signal from the control circuit 103. Up-sample to the same size to generate difference image data with the image data of the higher layer. The difference image data obtained by the first difference data generation circuit 403 is supplied to a second selector 402. The second selector 402 switches the output data to the output (difference image data) of the first difference data generation circuit 403 according to the encoding control signal from the control circuit 103. Therefore, the second blocking processing circuit 1
08 is supplied with difference image data.

As described above, the second data generation circuit 1
The difference image data of the higher layer supplied from 06 to the second blocking processing circuit 108 is blocked by the second blocking processing circuit 108 and the second coding circuit 11
At 0, a predetermined encoding process is performed for each block independently of the image of the base layer, and this is supplied to the bit stream generation circuit 111.

The bit stream generation circuit 111
A header corresponding to a predetermined application (transmission, storage, etc.) is added to the image data of the base layer from the encoding circuit 109 and the image data of the higher layer (differential image data) from the second encoding circuit 110. Then, it is incorporated into one bit stream to form a scalable bit stream of image data and output to the outside.

(Time Scalability Use Mode)

When this mode is designated, the first mode is set in the same manner as when the space scalability use mode is designated.
Frame memory 102 and second frame memory 10
4 is supplied to the first blocking processing circuit 107 and the second blocking processing circuit 108 via the first data generation circuit 105 and the second data generation circuit 106. However, the operations of the first data generation circuit 105 and the second data generation circuit 106 are different from the operations when the above-described space scalability use mode is specified.

That is, the first data generation circuit 105
(See FIG. 3 above), the first selector 301
The encoding control signal from the control circuit 103 (external information 112
The output destination is switched to the second selector 303 in accordance with the time scalability use mode designation control signal based on the first frame memory 102, and the YCbCr data from the first frame memory 102 is output. The second selector 303 controls the control circuit 1
03, the first selector 30
The YCbCr data from 1 is supplied to the frame rate controller 305. Frame rate controller 30
Reference numeral 5 denotes a frame-based downsample (reduces the resolution in the time direction of image data) for the YCbCr data from the second selector 301 according to the frame rate information included in the encoding control signal from the control circuit 103. Then, the image data of the base layer is generated. The image data of the base layer obtained by the frame rate controller 305 is supplied to the third selector 302. The third selector 302 switches the output data to the output (image data of the base layer) of the frame rate controller 305 according to the encoding control signal from the control circuit 103. Therefore, the first block processing circuit 107 is supplied with the image data of the base layer.
The image data of the basic layer is also supplied to a second data generation circuit 106 described later.

Thus, the first data generation circuit 1
The image data of the base layer supplied to the first block processing circuit 107 from step 05 is divided into blocks by the first block processing circuit 107, and the first coding circuit 109 performs predetermined block unit coding. The bit stream is supplied to the bit stream generation circuit 111.

On the other hand, in the second data generation circuit 106 (see FIG. 4), the first selector 401 sends the output destination to the second difference data generation circuit 404 according to the encoding control signal from the control circuit 103. By switching, the YCbCr data from the second frame memory 104 is output. At the same time, the frame memory 405 stores the control circuit 103
The first data generation circuit 1
The image data of the base layer from 05 is supplied to the second difference data generation circuit 404. The second difference data generation circuit 404 uses the image data of the base layer from the frame memory 405 as prediction information of the higher layer according to the encoding control signal from the control circuit 103, and calculates the temporally future and past image data. To generate difference image data as a higher layer. The difference image data obtained by the second difference data generation circuit 404 is
02. The second selector 402 switches the output data to the output (difference image data) of the second difference data generation circuit 404 according to the encoding control signal from the control circuit 103. Therefore, difference image data is supplied to the second blocking processing circuit 108.

Thus, the second data generation circuit 1
The difference image data of the higher layer supplied from 06 to the second blocking processing circuit 108 is blocked by the second blocking processing circuit 108 and the second coding circuit 11
At 0, a predetermined encoding process is performed for each block independently of the image of the base layer, and this is supplied to the bit stream generation circuit 111.

The bit stream generation circuit 111 transmits the image data of the base layer from the first encoding circuit 109 and the high-order layer data from the second encoding circuit 110 in the same manner as when the spatial scalability use mode is specified. A header is added to the image data (differential image data) to form a scalable bit-bit stream of image data, which is output to the outside.

(SNR Scalability Use Mode)

Each of the first frame memory 102 and the second frame memory 104 has an R / W control signal (control signal for designating an SNR scalability use mode based on external information) from the control circuit 103 (specifically, the CPU 701). ), The read / write operation of the YCbCr data from the conversion circuit 101 is performed. Thereby, in this case, the first frame memory 102 and the second frame memory 1
04 is read directly from the first
Are supplied to the second blocking processing circuit 107 and the second blocking processing circuit 108. Therefore, its YCbCr
The data is blocked by a first blocking processing circuit 107 and a second blocking processing circuit 108 and supplied to a first coding circuit 109 and a second coding circuit 110.

The first encoding circuit 109 includes the control circuit 10
In accordance with the encoding control signal from the third block, the YCbCr data from the first blocking processing circuit 107 is subjected to a predetermined encoding process in block units to generate encoded base layer image data. At this time, an encoding process is performed so as to have a predetermined code amount (compression rate) according to the encoding control signal. The coded base layer image data obtained by the first coding circuit 109 is supplied to the bit stream generation circuit 111, and the second base data is used as reference data in the coding processing of the higher layer image data. It is also supplied to the encoding circuit 110.

The second encoding circuit 110 includes the control circuit 10
3, the image data of the base layer from the first encoding circuit 109 is referred to future and past image data as prediction information of the higher layer by the encoding control signal from
The difference image data as an encoded higher layer is generated. The coded difference image data as the higher layer obtained by the second coding circuit 110 is supplied to the bit stream generation circuit 111.

The bit stream generation circuit 111 transmits the image data of the base layer from the first encoding circuit 109 and the image data of the second encoding circuit 110 from the second encoding circuit 110 in the same manner as when the spatial or temporal scalability use mode is specified. A header is added to the image data (differential image data) of the higher layer to form a scalable bit-bit stream of image data, which is output to the outside.

(Scalability function non-use mode)

Each of the first frame memory 102 and the second frame memory 104 responds to an R / W control signal (a control signal based on the scalability function non-use mode) from the control circuit 103 (specifically, the CPU 701).
The read / write operation of the YCbCr data from the conversion circuit 101 is performed. Thereby, in this case, the YCbCr data read from the first frame memory 102 and the second frame memory 104 is directly sent to the first block processing circuit 107 and the second block processing circuit 108. Supplied. Therefore, the YCbCr data is
Are divided into blocks by the block processing circuit 107 and the second block processing circuit 108, and the first encoding circuit 10
9 and the second encoding circuit 110 perform predetermined encoding processing in block units, and this is supplied to the bit stream generation circuit 111.

The bit stream generation circuit 111
A predetermined application (transmission, storage, etc.) is applied to each data from the encoding circuit 109 and the second encoding circuit 110.
To form a bit stream of image data and output it to the outside.

(Second Embodiment)

The present invention is applied to, for example, a decoding device 200 as shown in FIG. This decoding device 200 corresponds to the encoding device 100 in the first embodiment described above. That is, the decoding device 200 basically performs the reverse process of the encoding device 100. In particular, the decoding device 200 is configured to receive information (user information) from a user described later. It has been done.
The user information includes various information such as the image quality and the capability of the decoding device 200. Therefore, when the user of the decoding device 200 inputs various kinds of information in decoding, the external output information 212 according to the input is generated by the control circuit 208, and the external output information 212 is encoded as the above-described external information 112. Supplied to the gasifier 100.

FIG. 6 shows the internal configuration of the above-described control circuit 208, and FIG. 7 shows the internal configuration of the first data decoding circuit 209 of the decoding device 200. 8 is the first data decoding circuit 2 of the decoding device 200
09 shows the internal configuration.

Therefore, the above-mentioned user information is input by the following configuration. FIG. 9 shows the decoding device 2 of FIG.
1 shows a configuration of a system 240 having a function of “00”. The system 240 includes a monitor 241 and a personal computer (PC) as shown in FIG.
A main body 242 and a mouse 243 are provided, and each is connected to each other. The PC main body 242 has the function of the decoding device 200 having the configuration as shown in FIG.

Therefore, in the system 240, first,
On the monitor 241, a genre selection menu screen of selectable software (moving image) is displayed. For example, selection menu screens such as “movie”, “music”, “Photo”, and “etc.” are displayed. The user has mouse 2
By operating 43, a desired genre of software is designated on the screen displayed on the monitor 241.
Specifically, for example, the mouse cursor 244 is set to the desired software genre (“movie” in FIG. 9), and the mouse 243 is clicked or double-clicked. As a result, the genre (“movie”) is designated.

When such an operation is completed, next, FIG.
As shown at 0, a title menu screen of individual software corresponding to the genre (“movie”) selected on the selection menu screen of FIG. 9 is displayed. For example, “title-A”, “title-B”,
Title menus such as "title-C" and "title-D" are displayed. The user operates the mouse 243 to specify a desired title on the screen displayed on the monitor 241. Specifically, for example, for example, a desired title (“title
-A "), the mouse cursor 244 is set, and the mouse 2
43 Click operation or double click operation. Thus, the title (“title-A”) is designated.

When such an operation is completed, next, FIG.
As shown in FIG. 1, a condition setting screen for setting various decoding conditions is displayed for the title (“title-A”) selected on the title menu screen of FIG. Here, the following various conditions can be set. S / N: Low image quality (Low), high image quality (High), and optimal image quality (Aut
o) Specify one of: Frame Rate: Specifies one of low frame rate (Low), high frame rate (Full), and optimal frame rate (Auto) according to the decoding capability of this system. Full Spec: Encoding device (encoding device 10 in FIG. 1 above)
Designate the highest image quality (high code amount) image on the 0) side. Full Auto: Set various optimum conditions according to the decoding capability of this system.

Therefore, as shown in FIG. 11, the user sets the condition to be set (“S / N” in FIG. 11).
Then, the mouse cursor 244 is set, and the mouse 243 is clicked or double-clicked. This allows
As shown in FIG. 12, detailed conditions for setting "S / N" are displayed. That is, "Low", "Hig"
h "and" Auto ". The user positions the mouse cursor 244 on the condition (“Auto” in FIG. 12) desired to be set to “S / N” from the detailed conditions, and clicks or double-clicks the mouse 243. As a result, in FIG.
As for “S / N”, “Auto” (optimal image quality according to the decoding capability of the present system) is set.

As described above, information on various conditions set on the screen is supplied as external information 112 to the encoding device 100 according to the first embodiment described above. As described above, the encoding device 100 interprets the external information 112, selects the optimal scalability, determines various setting conditions (image size, compression ratio, and the like) in the scalability, and performs a series of operations. The encoding process is performed, and the result is output to the system 240 (decoding device 200) in FIG.

An object of the present invention is to supply a storage medium storing program codes of software for realizing the functions of the host and the terminal of each of the above-described embodiments to a system or an apparatus, and to provide a computer for the system or the apparatus. Needless to say, this can also be achieved by a program (or CPU or MPU) reading and executing a program code stored in a storage medium. In this case, the program code itself read from the storage medium realizes the function of each embodiment, and the storage medium storing the program code constitutes the present invention.

The storage media for supplying the program codes include ROM, floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, and C-ROM.
DR, a magnetic tape, a nonvolatile memory card, or the like can be used.

The functions of each embodiment are realized by executing the program code read by the computer, and the OS or the like running on the computer is executed based on the instructions of the program code. It goes without saying that a part or all of the actual processing is performed, and the function of each embodiment is realized by the processing.

Further, after the program code read from the storage medium is written in a memory provided in an extension function board inserted in the computer or a function extension unit connected to the computer, based on the instruction of the program code, The CPU provided in the function expansion board or function expansion unit performs part or all of the actual processing,
It goes without saying that the functions of the embodiments are realized by the processing.

[0096]

As described above, according to the present invention, on the encoding side, information on the decoding side (the capability of the decoding side, the resolution and code amount specified by the user on the decoding side, and the information between the encoding side and the decoding side). The encoding of the image data is performed according to the transmission line capacity. Specifically, the encoding side is provided with a unit (input unit) for receiving the performance and transmission line information of the decoding side, which is the output destination, or information desired by the user.
A scalable image suitable for the information is selected, encoded and transmitted in real time. On the other hand, the decoding side is provided with means (output means) for transmitting the processing capability of the decoding side, user information input by a user interface (designating means) and the like to the encoding side. With this configuration, even if the image desired by the decoding side (user on the decoding side) is any image, encoded image data suitable for the image can be provided to the decoding side. Also, it is possible to provide the decoding side with coded image data that is compatible with the line capacity used for image transmission and the performance of the decoding side (ensures compatibility). Therefore, a good decoded image can be obtained on the decoding side.

Further, if an interactive communication function (transmission means) is provided between the encoding side and the decoding side, it is possible to decode encoded image data suitable for any circumstances existing between the encoding side and the decoding side. Can be given to the side.
As a result, advantages such as broad compatibility between the encoding side and the decoding side and effective use of the line capacity can be provided. Furthermore, a user interface (designating means) is provided on the decoding side.
Is provided, it is possible to provide coded image data that flexibly adapts to the user's desire by using an interactive communication function (transmission means).

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an encoding device to which the present invention has been applied in a first embodiment.

FIG. 2 is a block diagram showing an internal configuration of a control circuit of the encoding device.

FIG. 3 is a block diagram showing an internal configuration of a first data generation circuit of the encoding device.

FIG. 4 is a block diagram showing an internal configuration of a second data generation circuit of the encoding device.

FIG. 5 is a block diagram illustrating a configuration of a decoding device to which the present invention has been applied in the second embodiment.

FIG. 6 is a block diagram showing an internal configuration of a control circuit of the decoding device.

FIG. 7 is a block diagram showing an internal configuration of a first data decoding circuit of the decoding device.

FIG. 8 is a block diagram showing an internal configuration of a second data decoding circuit of the decoding device.

FIG. 9 is a diagram for explaining a system having the function of the decoding device.

FIG. 10 is a diagram for explaining an operation of selecting a title of a genre menu on a monitor screen of the system.

FIG. 11 is a diagram illustrating a condition setting screen displayed by selecting the title.

FIG. 12 is a diagram for explaining an operation of setting a desired condition for a condition item selected on the condition setting screen.

FIG. 13 is a diagram for describing spatial scalability.

FIG. 14 is a diagram for explaining time scalability.

FIG. 15 is a diagram for explaining SNR scalability.

FIG. 16 is a block diagram illustrating a configuration of a conventional encoding device.

FIG. 17 is a block diagram illustrating a configuration of a conventional decoding device.

[Explanation of symbols]

 REFERENCE SIGNS LIST 100 Encoding device 101 Conversion circuit 102 First frame memory 103 Control circuit 104 Second frame memory 105 First data generation circuit 106 Second data generation circuit 107 First block processing circuit 108 Second block Processing circuit 109 First encoding circuit 110 Second encoding circuit 111 Bit stream generation circuit 112 External information

Claims (20)

[Claims] An encoding apparatus for generating at least two image data having different resolutions from the same image, encoding the image data according to a predetermined encoding method, and outputting the encoded image data. An encoding device comprising: input means for performing encoding; and control means for controlling an encoding operation in accordance with input information from the input means. 2. The encoding apparatus according to claim 1, wherein the image data having different resolutions includes image data having different spatial or temporal resolutions. 3. The encoding apparatus according to claim 1, wherein the input information includes at least one of the resolution information and the decoding side capability information as the output destination. 4. A decoding device for generating at least two image data having different resolutions from the same image and decoding the image data obtained by encoding the image data according to a predetermined encoding method, A decoding device comprising: a designation unit for designating a resolution; and an output unit for externally outputting designation information by the designation unit. 5. The decoding device according to claim 4, wherein the image data having different resolutions includes image data having different spatial or temporal resolutions. 6. The decoding device according to claim 4, wherein said specifying means specifies capability information of the own device. 7. An encoding unit for generating at least two image data having different resolutions from the same image, encoding these image data in accordance with a predetermined encoding system, and outputting the encoded data, and decoding an output of the encoding unit. An image processing apparatus comprising: a decoding unit; and the decoding unit includes at least a designation unit that designates the resolution in the encoding unit. 8. The image processing apparatus according to claim 7, wherein the image data having different resolutions includes image data having different spatial or temporal resolutions. 9. The image processing apparatus according to claim 7, wherein said specifying means specifies capability information of the own apparatus. 10. The decoding means comprises: first decoding means for decoding image data of an arbitrary resolution included in the output of the encoding means to obtain a decoded image; and at least one decoding means included in the output of the encoding means. 8. The image processing apparatus according to claim 7, further comprising: a second decoding unit that obtains a decoded image having an arbitrary resolution by decoding and adding two pieces of image data having different resolutions. 11. Encoding means for generating image data having at least two different code amounts from the same image, encoding these image data in accordance with a predetermined encoding method, and outputting the encoded data, and decoding the output of said encoding means. An image processing apparatus comprising: a decoding unit configured to specify at least the code amount in the encoding unit. 12. The setting means for setting at least one of the image data size and the frame rate corresponding to infrastructure information including at least one of the decoding capability and the transmission line capacity of the decoding means. The image processing apparatus according to claim 11, further comprising: 13. The decoding means includes: first decoding means for decoding image data having an arbitrary code amount included in an output of the encoding means to obtain a decoded image; and decoding means included in an output of the encoding means. The image processing apparatus according to claim 11, further comprising: a second decoding unit configured to decode and add at least two pieces of image data having different code amounts to obtain a decoded image having an arbitrary resolution. 14. An encoding unit for generating at least two image data having different code amounts from the same image, encoding these image data according to a predetermined encoding method, and outputting the encoded image data, and decoding an output of the encoding unit. An image processing apparatus comprising: a decoding unit that automatically transmits information of the decoding unit between the encoding unit and the decoding unit. The encoding unit includes: An image processing apparatus, which encodes and outputs the image data according to information sent via a means. 15. The image processing apparatus according to claim 14, wherein said encoding means determines a code amount at the time of encoding according to information transmitted via said transmission means. 16. An image processing apparatus according to claim 14, wherein said decoding means selects and decodes image data having an arbitrary code amount from each image data included in the output of said encoding means. 17. Encoding means for generating at least two image data having different code amounts from the same image, encoding these image data according to a predetermined encoding method, and outputting the encoded data, and decoding the output of the encoding means. An image processing apparatus comprising: a decoding unit configured to transmit information of the decoding unit between the encoding unit and the decoding unit. 18. The image processing apparatus according to claim 17, wherein the information of the decoding unit includes at least a code amount that can be transmitted by the decoding unit. 19. The image processing apparatus according to claim 17, wherein said decoding means selects and decodes image data having an arbitrary code amount from each image data included in the output of said encoding means. 20. An encoding device according to any one of claims 1 to 3, a decoding device according to any one of claims 4 to 6, and an image processing device according to any one of claims 7 to 19. An image processing system comprising: