JP2003244694A - Information processing device and method, recording medium and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To appropriately distribute data in the case of performing hierarchical encoding based on a temporal scalability function. <P>SOLUTION: An encoding part 15 applies DCT processing, quantization processing, encoding processing or the like to a base layer inputted from a frame processing part 13, however, decides a quantization step Q-ba in accordance with a requested bit rate and performs quantization processing according to the decided quantization step Q-ba. The decided quantization step Q-ba is appropriately supplied to an encoding part 16. The encoding part 16 applies DCT processing, quantization processing, encoding processing or the like to an enhancement layer inputted from a frame processing part 14, and however, calculates a quantization step Q-en on the basis of the quantization step Q-ba supplied from the encoding part 15 in performing quantization processing. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus and method, a recording medium, and a program, and more particularly, to an information processing apparatus and method capable of performing proper data distribution by performing hierarchical coding. The present invention relates to a recording medium and a program.

[0002]

2. Description of the Related Art According to a time scalability function provided in MPEG2 or MPEG4, image data is transmitted, for example, to a base layer of data transmitted at a low frame rate and an enhancement layer of data transmitted at a high frame rate. It can be encoded for each layer.

As a result, the encoded base layer data (base layer bit stream) is transmitted to the client device using the narrow transmission band, and the base layer bit stream and the encoded enhancement layer data (enhancement) are transmitted. Since both of the layer bitstreams) can be transmitted to a client device that uses a wide transmission band, data distribution can be performed according to the transmission capability of the client.

In the client device which has received the base layer bitstream, the bitstream is decoded as it is, so that a low frame rate image is reproduced. On the other hand, in the client device that receives the delivery of the enhancement layer bitstream together with the base layer bitstream, the base layer bitstream is interpolated and decoded by the enhancement layer bitstream, so that a high frame rate image is reproduced.

[0005]

By the way, also in the hierarchical coding, the motion compensation and the DCT are performed in the same manner as the normal coding.
Encoding based on (discrete cosine transform) is performed. That is, the quantization is performed based on the quantization step determined according to a predetermined rule.

Therefore, when the base layer and the enhancement layer are encoded based on the quantization step determined by one method, almost uniform image quality can be obtained. In this case, the base layer bit rate is variable. The bit rate is set, and as a result, the variable bit rate data is distributed to the client device that uses the narrow transmission band that requires the fixed bit rate.

Further, if the quantization step is individually determined for each layer, data of a fixed bit rate can be transmitted to a narrow transmission band target, but in this case, a base close in time is used. The image quality of the layer bitstream and that of the enhancement layer bitstream may differ greatly. At this time, if the base layer bitstream is interpolated by the enhancement layer bitstream, an unnatural image is reproduced. That is, there is a case where proper reproduction is not performed in the client device for the wide transmission band.

As described above, in the conventional case, there is a problem in that proper data distribution cannot be performed to the client when the hierarchical encoding is performed.

The present invention has been made in view of such a situation, and is to enable appropriate data distribution by performing hierarchical coding based on the time scalability function.

[0010]

An information processing apparatus according to the present invention is a first layer of a first layer of image data having a layered structure.
First data is thinned out in order to be transmitted at a predetermined transmission rate, and the first data is quantized and coded based on the first quantization step, and the second hierarchical structure is used. Of the second data of the hierarchy of 2), the second coding means for quantizing and coding the second data based on the second quantization step corresponding to the first quantization step, and only the coded first data. Is provided, and second output means for outputting both the encoded first data and the encoded second data are provided.

The second encoding means quantizes and encodes the first data decimated by the first encoding means together with the second data based on the second quantization step. Is characterized by.

According to the information processing method of the present invention, in order to transmit the first data of the first layer of the image data having a hierarchical structure at a predetermined transmission rate, the first data is thinned out and the first data is removed. And a second quantization step corresponding to the first quantization step in which the first data is quantized and coded based on the first quantization step and the second data of the second layer of the hierarchical structure is A second encoding step of quantizing and encoding based on the above, a first output step of outputting only the encoded first data, the encoded first data and the encoded first data. And a second output step for outputting both of the two data.

A recording medium program according to the present invention includes a first data thinning operation for transmitting first data of a first hierarchy of image data having a hierarchical structure at a predetermined transmission rate, and a first data thinning operation. And a second quantum corresponding to the first quantization step of the second data of the second layer of the hierarchical structure, the first quantization control step controlling quantization and encoding based on the quantization step of A second encoding control step for controlling the quantization and encoding based on the encoding step, a first output control step for controlling the output of only the encoded first data, and the encoded first A second output step for controlling the output of both the data and the encoded second data.

The program of the present invention is a thinning-out of the first data and a first quantization of the image data having a hierarchical structure for transmitting the first data of the first hierarchy at a predetermined transmission rate. A first encoding control step for controlling the step-based quantization and encoding, and a second quantization step corresponding to the first quantization step of the second data of the second layer of the hierarchical structure. A second encoding control step for controlling the quantization and the encoding based on the encoded first
A first output control step for controlling the output of only the data of the above, and the encoded first data and the encoded second
A second output step for controlling the output of both of the data in the computer is executed by the computer.

In the information processing apparatus and method and the program of the present invention, the first data of the image data having the hierarchical structure is transmitted in order to transmit the first data of the first layer at a predetermined transmission rate. While being thinned out,
The second data of the second layer of the hierarchical structure is quantized and encoded based on the first quantization step, and the second data of the second layer of the hierarchical structure is quantized based on the second quantization step corresponding to the first quantization step. Only the encoded first data that is encoded and encoded is output, and both the encoded first data and the encoded second data are output.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an example of use of an encoding device 1 to which the present invention is applied.

The encoding apparatus 1 distributes image data into a base layer, which is a layer of data transmitted at a low frame rate, and an enhancement layer, which is a layer of data transmitted at a high frame rate, and each layer is divided into layers. The data is encoded according to the MPEG4 standard. The encoding device 1 transmits the base layer bitstream and the enhancement layer bitstream obtained as a result of the hierarchical encoding to the distribution server 2.

Here, the MPEG4 encoding will be briefly described. In MPEG4, an input image is divided into a shape (Shape) of a target video object and its picture (Texture) and encoded. The unit of this video data is VOP (Vide
o Object Plane).

For example, when an image is composed of two objects, a person and a background, each frame has two VOs.
It is divided into P and encoded. The information that makes up each VOP is
These are shape information, motion information, and texture information of the object. In this way, by introducing the VOP structure, when the image to be processed is composed of a plurality of video objects, this can be divided into a plurality of VOPs and individually coded.

There are three types of VOP prediction methods: intra-frame coding, forward prediction, and bidirectional prediction. Prediction unit is 16 ×
It is a 16-pixel macroblock (MB).

For example, the bidirectional prediction is a method of predicting the current VOP from both directions of the past VOP and the future VOP, like the B picture of MPEG2.

Returning to FIG. 1, the distribution server 2 sends the base layer bit stream and the enhancement layer bit stream from the encoding device 1 to the wide transmission band target decoding device 3 or the narrow transmission band target according to the transmission capability of the client. Deliver to the decryption device 4.

Specifically, the distribution server 2 multiplexes and distributes the base layer bit stream and the enhancement layer bit stream to the wide transmission band target decoding device 3 and distributes them to the narrow transmission band target decoding device 4. Deliver the stream.

The wide transmission band target decoding apparatus 3 interpolates and decodes the base layer bitstream with the enhancement layer bitstream to reproduce a high frame rate image. The narrow transmission band target decoding device 4 is the distribution server 2
It decodes the base layer bitstream distributed from and plays back low frame rate images.

FIG. 2 shows an example of the functional configuration of the encoding device 1.

The separation unit 11 determines a layer to which a VOP forming an input image belongs, and supplies each VOP and setting information indicating the layer to which the VOP belongs (information indicating a base layer or an enhancement layer) to the rearrangement unit 12. .

The sorting section 12 is supplied from the separating section 11.
The VOP is set in the layer according to the setting information similarly supplied from the separating unit 11, and the hierarchical structure of the base layer and the enhancement layer is formed. For this example, in-plane coded VOP (I-VOP) and forward predicted VOP (P-VOP)
Is set in the base layer and bidirectionally predicted VOP (B-
VOP) is set in the enhancement layer.

The sorting section 12 also sets the VO set for each layer.
The Ps are rearranged in the encoding order determined by the type of VOP, and the base layer is supplied to the frame processing unit 13 and the enhancement layer is supplied to the frame processing unit 14.

For example, as shown in FIG. 3A, I,
When the images displayed in the order indicated by the numbers following P and B are input to the rearrangement unit 12, the rearrangement unit 12 refers to the VOP corresponding to the start point of the arrow in FIG. The order of the VOPs of the base layer and the enhancement layer is rearranged so that the VOP corresponding to the end point is predicted.

The frame processing section 13 is, for example, a VO set in the base layer according to the required bit rate.
An appropriate VOP is extracted from P, and the extracted VOP is supplied to the frame processing unit 14. The frame processing unit 13 also
The base layer from which the predetermined VOP is extracted is supplied to the encoding unit 15.

The frame processing unit 14 is the frame processing unit 1.
VOP supplied from 3 (V extracted from the base layer
OP) is incorporated into the corresponding position of the enhancement layer and is supplied to the encoding unit 16.

In the case of the image of FIG. 3A, the frame processing unit 13
Removes P-VOP (P3) from the base layer,
When supplied to the frame processing unit 14, the frame processing unit 1
4 incorporates the P-VOP (P3) in the enhancement layer so that it is encoded in the order shown by the arrow in FIG. 3B.

It is also possible to discard the P-VOP (P3) extracted from the base layer by the frame processing unit 13 so that it is not incorporated in the enhancement layer as shown in FIG. 3C.
Since the B-VOP (B1) and B-VOP (B2) are coded with a delay longer than usual (because they are coded after the P-VOP (P6) is coded), image quality degradation is expected. It Then, as shown in FIG. 3D, from the enhancement layer
B-VOP (B1) and B-VOP (B2) can be deleted, but in this case, the transmission amount is reduced accordingly, so that the transmission capacity of the wide-bandwidth target decoding device 3 can be fully exerted. Disappear.

Returning to FIG. 2, the encoding unit 15 applies DC to the base layer input from the frame processing unit 13.
T processing, quantization processing, encoding processing, etc. are performed, but the quantization step Q_ba is determined according to the required bit rate, and the quantization processing is performed according to the determined quantization step Q_ba. The encoding unit 15 uses the determined quantization step.
Q_ba is appropriately supplied to the encoding unit 16.

The encoding unit 15 transmits the encoded base layer bitstream to the distribution server 2.

The encoding unit 16 includes a frame processing unit 14
DCT for the enhancement layer input from
Although the processing, the quantization processing, the encoding processing, and the like are performed, when the quantization processing is performed, the quantization step Q_en is calculated based on the quantization step Q_ba supplied from the encoding unit 15.

Specifically, the equation (1) is calculated. Q_en
= (SAD_prev + SAD_next) / (2 × SAD_en) × a × Q_en_0
... (1)

In the equation, Q_en_0 is obtained by the equation (2). That is, in consideration of the smoothness of the entire sequence of the base layer and the enhancement layer, Q_en_0, as shown in FIG.
The quantization step of VOP_ba_prev is Q_ba_prev as the quantization step of VOP_en of the enhancement layer separated by
And from the quantization step Q_ba_next of VOP_ba_next,
Interpolated. Q_en_0 = (Q_ba_prev × d2 + Q_b
a_next × d1) / (d1 + d2) ・ ・ ・ (2)

In addition, SAD (sum of absolute di in equation (1)
fference) _prev and SAD_next, and SAD_en are V
A value that expresses the complexity of OP, which is ME (Motion Estimation)
The absolute value of the difference between the pixels found by matching the reference images in the search range in the pixel block.
It is the sum of AD in all pixel blocks in that VOP.

Therefore, Q_en changes Q_en_0 to Q_en_0 and VO.
Considering the complexity of P, VOP_ba_prev and VO of base layer
Average of SAD_prev and SAD_next of P_ba_next ((SAD_prev + S
AD_next) / 2) of the enhancement layer quantized with
It is corrected by multiplying the ratio of SAD (SAD_en) of VOP and a predetermined coefficient a.

The encoding unit 16 transmits the encoded enhancement layer bitstream to the distribution server 2.

As described above, the coding of the base layer and the coding of the enhancement layer are separately performed, and the quantization step Q_en used when coding the VOP of the enhancement layer is corresponded. Since the quantization step Q_ba used when encoding the VOP of the base layer is used, data of a fixed bit rate can be transmitted to the narrow transmission band target decoding device 4, and the time Since the image quality of the base layer bitstream and the enhancement layer bitstream that are close to each other can be prevented from being significantly different,
Data to be reproduced appropriately can be distributed to the wide transmission band target decoding device 3.

Next, the operation of the rearrangement unit 12 corresponding to the scene change of the input image will be described.

For example, in the image of FIG. 5A, if there is a scene change in the B-VOP (B4) of the enhancement layer displayed fifth, the rearrangement unit 12 detects the scene change and As shown in FIG. 5, the enhancement layer B-VOP (B4) and the P-VOP (P6) after the scene change and the P-VOP (P3) before the scene change in FIG. 5A are used to be predicted.
B-VOP (B5) is changed to P-VOP (P4) and P-VOP (P5), and P-VOP (P6) (Fig. 5A) of the base layer is changed to I-VOP
Change to (I6) (Fig. 5B). And the sorting section 12
Enhancement layer P-VOP with sea change
The VOPs are rearranged so that (P4) and the subsequent P-VOP (P5) are predicted from the base layer I-VOP (I6) after the sea change.

FIG. 5C shows the processing procedure of the rearrangement unit 12 when there is a scene change in the B-VOP (B5) of the enhancement layer displayed sixth in the image of FIG. 5A. As shown in FIG. 5C, the enhancement layer B-VOP (B4) is predicted by referring to the P-VOP (P6) after the scene change and the P-VOP (P3) before the scene change in FIG. 5A. ) And B-VOP
(B5) is changed to P-VOP (P4) and P-VOP (P5),
Base layer P-VOP (P6) (Fig. 5A) is I-VOP (I6)
(FIG. 5C). And as shown in FIG. 5C,
The enhancement layer P-VOP (P4) is predicted only from the base layer P-VOP (P3) before the chain change, and the enhancement layer P-VOP (P5) after the sea change is after the sea change. The VOPs are rearranged so as to be predicted only from the base layer I-VOP (I6).

As shown in FIG. 5C, the enhancement layer B-VOP (B7) and B-VOP (B8) are predicted from the base layer I-VOP (I6) and P-VOP (P9). Like

Normally, a VOP that has a scene change is IV
Although it is considered as an OP, the B-VOP of the enhancement layer, which had such a scene change, was changed to the P-VOP and it was made to be predicted from the VOP of the base layer after the sea change. It is possible to prevent a reduction in coding efficiency due to the presence of I-VOP in both the enhancement layer and the enhancement layer, and to appropriately reproduce an image.

FIG. 5D shows the processing procedure of the rearrangement unit 12 when there is a scene change in the P-VOP (P6) of the base layer, which is displayed seventh in the image of FIG. 5A. . As shown in FIG. 5D, the P-VOP (P6) having the scene change in FIG. 5A and the scene before the scene change
Enhancement layers B-VOP (B4) and B-VOP (B5) that are made to be predicted from P-VOP (P3) are P-VO
Changed to P (P4) and P-VOP (P5), P-of the base layer
VOP (P6) is changed to I-VOP (I6). Then, the VOPs are rearranged so that the P-VOPs (P4) and P-VOPs (P5) of the enhancement layer are predicted only from the P-VOPs (P3) of the base layer before the scene change.

As shown in FIG. 5D, the enhancement layer B-VOP (B7) and B-VOP (B8) are predicted from the base layer I-VOP (I6) and P-VOP (P9). Like

FIG. 6 shows an example of the hardware configuration of the portion of the encoding apparatus 1 relating to MPEG4 encoding. The processes of the separation unit 11 to the encoding unit 16 in FIG. 2 are performed by the memory 21 to the texture encoding unit 23 operating in cooperation with each other.

The input image data is supplied to the subtractor 22 and the motion predictor 27 via the memory 21 capable of temporarily storing the image data and storing a plurality of images in order to replace it in a predetermined order. It

The subtractor 22 will be described later with respect to the input image data.
The difference with the predicted image data between VOPs is taken, and the difference value is
The data is supplied to the texture encoder 23 as VOP data for which inter-VOP encoding, which will be described later, is performed.

The texture coding unit 23 uses a DCT (not shown).
It has a circuit, a quantizer, and a VLC (Variable Length Coding) circuit, and performs DCT processing on the image data from the difference unit 22.
Quantization processing and variable length coding processing are performed, and the resulting data (base layer bitstream and enhancement layer bitstream) is supplied to the multiplexer 29.

The texture coding unit 23 also has an inverse quantization circuit and an inverse DCT circuit, which dequantizes the quantized quantized coefficient into a DCT coefficient, and its D
The inverse DCT processing is performed on the CT coefficient, and the data obtained as a result is supplied to the adder 24.

Since the inter-VOP predicted image data output from the motion compensator 26 is also supplied to the adder 24, the adder 24
Adds these data and supplies it to the memory 25 for storage.

The motion predictor 27 detects, for each frame of the input image, a portion that is very similar to a macroblock from the frame that was input immediately before, and the portion and the relative macroblock relative to each other. A vector representing a shift in the physical positional relationship is detected as a motion vector. The detected motion vector is supplied to the motion compensator 26 and the multiplexer 29.

The motion compensator 26 motion-compensates the VOP stored in the memory 25 according to the motion vector from the motion predictor 27, and supplies the inter-VOP predicted image data obtained as a result to the differencer 22.

The shape encoder 28 encodes the shape information of the object (object) and supplies it to the multiplexer 29.

The multiplexer 29 appropriately multiplexes the encoded shape information from the shape encoder 28, the motion vector from the motion predictor 27, and the encoded image data from the texture encoder 23. The data obtained as a result is output via the buffer 30.

[0060]

According to the information processing apparatus and method and the program of the present invention, the first data of the first layer of the image data having the hierarchical structure is transmitted at a predetermined transmission rate. Data is quantized and encoded based on the first quantization step, and the second data of the second layer of the hierarchical structure is quantized by the second quantization corresponding to the first quantization step. Quantize and code based on steps, output only the coded first data, coded first data and coded second data
Since both of the above data are output, appropriate data distribution can be performed.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a usage example of an encoding device to which the present invention has been applied.

FIG. 2 is a block diagram showing a functional configuration example of the encoding device in FIG.

FIG. 3 is a diagram illustrating an operation of a rearrangement unit 12 in FIG.

FIG. 4 is a diagram illustrating a method of calculating a quantization step Q_en of a VOP of an enhancement layer.

FIG. 5 is a diagram illustrating an operation of the rearrangement unit 12 corresponding to a scene change.

FIG. 6 is a block diagram showing a configuration example of the encoding device in FIG. 1.

[Explanation of symbols]

1 encoding device, 2 distribution server, 3 wide transmission band target decoding device, 4 narrow transmission band target decoding device, 11
Separation unit, 12 rearrangement unit, 13 frame processing unit,
14 frame processing section, 15 encoding section, 16
Encoding section

Continued front page    F-term (reference) 5C059 LB07 MA00 MA05 MA14 MA23                       MA33 MB03 MB04 MB11 MB12                       MB14 MC11 MC31 NN23 NN43                       PP04 PP22 PP28 PP29 SS06                       SS20 TA03 TA46 TB00 TB17                       TC04 TC06 TC10 TC14 TC37                       TC42 TD03 TD05 TD06 TD16                       UA02 UA05 UA32 UA33 UA39                 5J064 AA01 BA13 BB03 BC01 BC08                       BC09 BC14 BC29 BD03

Claims (5)

[Claims] 1. In order to transmit the first data of the first layer of the image data having a hierarchical structure at a predetermined transmission rate, the first data is thinned out and the first quantization step is performed. A first encoding unit that quantizes and encodes based on the second data of the second layer of the hierarchical structure;
Second encoding means for performing quantization and encoding based on a second quantization step corresponding to the quantization step of, and first output means for outputting only the encoded first data, An information processing apparatus comprising: a second output unit that outputs both the encoded first data and the encoded second data. 2. The second encoding means, based on the second quantization step, together with the second data, the first data thinned by the first encoding means. The information processing apparatus according to claim 1, wherein the information processing apparatus is quantized and encoded. 3. In order to transmit the first data of the first layer of the image data having a hierarchical structure at a predetermined transmission rate, the first data is thinned out and the first quantization step is performed. A first encoding step of quantizing and encoding based on the first data, a second data of a second layer of the hierarchical structure,
A second encoding step of quantizing and encoding based on a second quantizing step corresponding to the quantizing step of :, a first output step of outputting only the encoded first data, A second output step of outputting both the encoded first data and the encoded second data. 4. The thinning of the first data for transmitting the first data of the first hierarchy at a predetermined transmission rate of the image data having a hierarchical structure, and based on a first quantization step. A first coding control step of controlling quantization and coding, and a first coding control step of the second data of the second layer of the hierarchical structure.
A second encoding control step for controlling quantization and encoding based on a second quantization step corresponding to the first quantization step of; and a first encoding step for controlling output of only the encoded first data. A computer-readable program, comprising: an output control step; and a second output step of controlling output of both the encoded first data and the encoded second data. A recording medium on which is recorded. 5. Based on the thinning of the first data for transmitting the first data of the first layer at a predetermined transmission rate of the image data having a hierarchical structure, and a first quantization step. A first coding control step of controlling quantization and coding, and a first coding control step of the second data of the second layer of the hierarchical structure.
A second encoding control step for controlling quantization and encoding based on a second quantization step corresponding to the first quantization step of; and a first encoding step for controlling output of only the encoded first data. A computer is caused to perform a process including an output control step and a second output step of controlling the output of both the encoded first data and the encoded second data. program.