JP4499204B2 - Image signal multiplexing apparatus and method, and transmission medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image signal multiplexing apparatus and method. , In particular, regarding data transmission media, data recorded on a recording medium such as a magneto-optical disk or magnetic tape, reproduced from the recording medium, and displayed on a display, a video conference system, a videophone system, a broadcasting device, An image signal multiplexing apparatus and method suitable for application to data that is transmitted from a transmission side to a reception side via a transmission path and displayed, edited, or recorded on the reception side, such as a media database search system , And a transmission medium.
[0002]
[Prior art]
For example, in a system that transmits a moving image signal to a remote place such as a video conference system and a video phone system, in order to efficiently use a transmission path, the line correlation of video signals and the correlation between frames are used. Compress and encode image signals Ru It is made like that.
[0003]
In recent years, since the processing capability of computers has improved, moving image information terminals using computers are becoming widespread. In such a system, information is transmitted to a remote place through a transmission line such as a network. In this case as well, signals such as image signals, sound signals, and data to be transmitted are compressed and transmitted in order to efficiently use the transmission path.
[0004]
On the terminal side, the compressed signal transmitted based on a predetermined method is decoded, the original image signal, sound signal, data, etc. are restored, and output to a display or a speaker provided in the terminal. In the conventional technology, the transmitted image signal or the like was only output to the display terminal as it is, but in the information terminal using a computer, after converting the plurality of image signals, acoustic signals and data, It has become possible to display in a two-dimensional or three-dimensional space. Such processing is performed by describing information in a two-dimensional and three-dimensional space by a predetermined method on the transmission side, and performing, for example, a predetermined conversion process for an image signal or the like according to the description by the terminal and displaying the information. Can be realized.
[0005]
As a typical description method of such spatial information, for example, there is VRML (Virtual Reality Modeling Language). This is also standardized in ISO-IEC / JTC1 / SC2 4 and the latest version of VRML2.0 is described in IS14772. VRML is a language for describing a three-dimensional space, and defines a collection of data for describing attributes and shapes of the three-dimensional space. This collection of data is called a node. In order to describe the three-dimensional space, it is described how these nodes that are defined in advance are combined. Nodes are defined to indicate attributes such as color and texture, and those indicating the shape of a polygon.
[0006]
In an information terminal using a computer, a predetermined object is generated using a polygon or the like by CG (Computer Graphics) in accordance with the description such as VRML. In VRML, a texture can be pasted on a three-dimensional object composed of polygons generated in this way. If the texture to be pasted is a still image, a node called Texture is defined. If it is a moving image, a node called MovieTexture is defined. Information about the texture to be pasted to this node (file name, display start time, display end time, etc.) Are listed.
[0007]
Here, the pasting of the texture (hereinafter referred to as texture mapping as appropriate) will be described with reference to FIG. First, a texture (image signal) to be pasted, a signal indicating its transparency (key signal), and three-dimensional object information are input from the outside and stored in a predetermined storage area of the memory group 151. The texture is stored in the texture memory 152, the signal indicating the transparency is stored in the gray scale memory 153, and the three-dimensional object information is stored in the three-dimensional information memory 154. Here, the three-dimensional object information includes polygon formation information and illumination information.
[0008]
The rendering circuit 155 forms a three-dimensional object with polygons based on predetermined three-dimensional object information recorded in the memory group 151. The rendering circuit 155 reads a signal indicating a predetermined texture and transparency from the memory 152 and the memory 153 based on the three-dimensional object information, and pastes the signal to the generated three-dimensional object. The signal indicating the transparency indicates the transparency of the texture at the corresponding position, and indicates the transparency of the object at the position where the texture at the corresponding position is pasted. The rendering circuit 155 supplies the signal of the object to which the texture is pasted to the two-dimensional conversion circuit 156.
[0009]
The two-dimensional conversion circuit 156 converts a three-dimensional object into a two-dimensional image signal obtained by mapping a three-dimensional object onto a two-dimensional plane based on viewpoint information supplied from the outside. The three-dimensional object converted into the two-dimensional image signal is further output to the outside. The texture may be a still image or a moving image. In the case of a moving image, the above operation is performed each time the image frame of the moving image to be pasted is changed.
[0010]
In VRML, JPEG (Joint Photographic Experts Group), which is one of the high-efficiency encoding methods for still images, and MPEG (Moving Picture Experts Group), which is one of the moving image encoding methods, are used as the texture format to be pasted. Compressed image formats are also supported. In this case, the texture (image) is decoded by a decoding process based on a predetermined compression method, and the reverse image signal is recorded in the memory 152 in the memory group 151.
[0011]
The rendering circuit 155 pastes the texture recorded in the memory 152 regardless of the format of the image, whether it is a moving image or a still image, and the content thereof. One texture that can be pasted to one polygon is always one texture stored in the memory, and a plurality of textures cannot be pasted to one polygon.
[0012]
By the way, when transmitting such three-dimensional information and texture information via a transmission line, it is necessary to compress and send the information in order to efficiently use the transmission line. In particular, when a moving image is pasted on a three-dimensional object, it is essential to compress and transmit the moving image.
[0013]
For example, the MPEG method described above was discussed in ISO-IEC / JTC1 / SC2 / WG11 and was proposed as a standard proposal. The method is adopted. In MPEG, several profiles and levels are defined to support various applications and functions. The most basic is the main profile main level (MP @ ML).
[0014]
A configuration example of an MPEG MP @ ML encoder will be described with reference to FIG. The input image signal is first input to the frame memory group 1 and stored in a predetermined order. Image data to be encoded is input to a motion vector detection circuit (ME) 2 in units of macroblocks. The motion vector detection circuit 2 processes the image data of each frame as an I picture, P picture, or B picture according to a predetermined sequence set in advance. It is predetermined (for example, I, B, P, B, P,..., B, P) that an image of each frame that is sequentially input is processed as an I, P, or B picture. Are processed in the order of
[0015]
The motion vector detection circuit 2 performs motion compensation with reference to a predetermined reference frame determined in advance, and detects the motion vector. There are three types of motion compensation (interframe prediction): forward prediction, backward prediction, and bidirectional prediction. The prediction mode for P pictures is only forward prediction, and the prediction modes for B pictures are three types: forward prediction, backward prediction, and bidirectional prediction. The motion vector detection circuit 2 selects a prediction mode that minimizes the prediction error, and generates a prediction vector at that time.
[0016]
At this time, for example, the prediction error is compared with the variance of the macroblock to be encoded. When the variance of the macroblock is smaller, the prediction is not performed on the macroblock, and the intraframe encoding is performed. In this case, the prediction mode is intra-picture coding (intra). The motion vector and the prediction mode are input to the variable length coding circuit 6 and the motion compensation circuit (MC) 12.
[0017]
The motion compensation circuit 12 generates predicted image data based on the input motion vector, and the predicted image data is input to the arithmetic circuit 3. The arithmetic circuit 3 calculates difference data between the value of the macroblock to be encoded and the value of the predicted image, and outputs it to the DCT circuit 4. In the case of an intra macroblock, the arithmetic circuit 3 outputs the macroblock signal to be encoded to the DCT circuit 4 as it is.
[0018]
In the DCT circuit 4, the input signal is subjected to DCT (Discrete Cosine Transform) processing and converted to DCT coefficients. This DCT coefficient is input to the quantization circuit (Q) 5, quantized in a quantization step corresponding to the data accumulation amount (buffer accumulation amount) of the transmission buffer 7, and then the quantized data is converted into a variable length coding circuit. (VLC) 6 is input.
[0019]
The variable length encoding circuit 6 corresponds to the quantization step (scale) supplied from the quantization circuit 5 and converts the quantized data (for example, I picture data) supplied from the quantization circuit 5 to, for example, Huffman. The data is converted into a variable length code such as a code and output to the transmission buffer 7. The variable length coding circuit 6 is also set with a quantization step (scale) by the quantization circuit 5 and a prediction mode (intra-picture prediction, forward prediction, backward prediction, or bidirectional prediction) set by the motion vector detection circuit 2. ) And a motion vector are input, and these are also variable-length encoded.
[0020]
The transmission buffer 7 temporarily stores the input encoded data, and outputs data corresponding to the storage amount to the quantization circuit 5. When the remaining amount of data increases to the allowable upper limit value, the transmission buffer 7 increases the quantization scale of the quantization circuit 5 by the quantization control signal, thereby reducing the data amount of the quantized data. On the other hand, when the remaining data amount is reduced to the allowable lower limit value, the transmission buffer 7 reduces the quantization scale of the quantization circuit 5 by the quantization control signal, thereby reducing the data amount of the quantized data. Increase. In this way, overflow or underflow of the transmission buffer 7 is prevented. The encoded data stored in the transmission buffer 7 is read at a predetermined timing, and is output as a bit stream to the transmission path.
[0021]
On the other hand, the quantized data output from the quantizing circuit 5 is input to the inverse quantizing circuit (IQ) 8 and is inversely quantized corresponding to the quantization step supplied from the quantizing circuit 5. Output data of the inverse quantization circuit 8 (DCT coefficient obtained by inverse quantization) is input to an IDCT (inverse DCT) circuit 9. The IDCT circuit 9 performs inverse DCT processing on the input DCT coefficient, and the obtained output data (difference data) is supplied to the arithmetic circuit 10. The arithmetic circuit 10 adds the difference data and the predicted image data from the motion compensation circuit 12, and the output image data is stored in the frame memory (FM) group 11. In the case of an intra macroblock, the arithmetic circuit 10 supplies the output data from the IDCT circuit 9 to the frame memory group 11 as it is.
[0022]
Next, a configuration example of an MPEG MP @ ML decoder will be described with reference to FIG. The encoded image data (bit stream) transmitted through the transmission path is received by a reception circuit (not shown), reproduced by a reproduction device, temporarily stored in the reception buffer 21, and then encoded. The data is supplied to the variable length decoding circuit (IVLC) 22 as data. The variable-length decoding circuit 22 performs variable-length decoding on the encoded data supplied from the reception buffer 21, the motion vector and prediction mode to the motion compensation circuit 27, and the quantization step to the inverse quantization circuit (IQ) 23. , And outputs the decoded quantized data to the inverse quantization circuit 23.
[0023]
The inverse quantization circuit 23 inversely quantizes the quantized data supplied from the variable length decoding circuit 22 according to the quantization step supplied from the variable length decoding circuit 22 and obtains output data (obtained by being inversely quantized). The obtained DCT coefficient) is output to the IDCT circuit 24. The output data (DCT coefficient) output from the inverse quantization circuit 23 is subjected to inverse DCT processing by the IDCT circuit 24, and the output data (difference data) is supplied to the arithmetic circuit 25.
[0024]
When the output data output from the IDCT circuit 24 is I picture data, the output data is output from the arithmetic circuit 25 as image data, and image data (P or B picture data) that is input to the arithmetic circuit 25 later. ) Is supplied to the frame memory group 26 and stored therein. The image data is output to the outside as it is as a reproduced image. When the data output from the IDCT circuit 24 is a P or B picture, the motion compensation circuit 27 is supplied from the variable length decoding circuit 22 and is stored in the frame memory group 26 according to the motion vector and the prediction mode. Predictive image data is generated from the data and output to the arithmetic circuit 25. The arithmetic circuit 25 adds the output data (difference data) input from the IDCT circuit 24 and the predicted image data supplied from the motion compensation circuit 27 to obtain output image data. In the case of a P picture, the output data of the arithmetic circuit 25 is also stored as predicted image data in the frame memory group 26 and used as a reference image of an image signal to be decoded next.
[0025]
In addition to MP @ ML, MPEG defines various profiles and levels and provides various tools. Scalability is one such tool. In addition, in MPEG, a scalable encoding method that realizes scalability corresponding to different image sizes and frame rates is introduced. For example, in the case of spatial scalability, when only a lower layer bitstream is decoded, a small image size image signal is decoded, and when a lower layer and upper layer bitstream is decoded, a large image size image signal is decoded. .
[0026]
The spatial scalability encoder will be described with reference to FIG. In the case of spatial scalability, the lower layer corresponds to an image signal having a small image size, and the upper layer corresponds to an image signal having a large image size.
[0027]
The lower layer image signal is first input to the frame memory group 1 and encoded in the same manner as MP @ ML. However, the output data of the arithmetic circuit 10 is not only supplied to the frame memory group 11 and used as the predicted image data of the lower layer, but also the same image size as the image size of the upper layer by the image enlarging circuit (up sampling) 31. After being enlarged to the upper layer, it is also used for the predicted image data of the upper layer.
[0028]
The upper layer image signal is first input to the frame memory group 51. The motion vector detection circuit 52 determines a motion vector and a prediction mode, similarly to MP @ ML. The motion compensation circuit 62 generates predicted image data according to the motion vector determined by the motion vector detection circuit 52 and the prediction mode, and outputs the predicted image data to the weight addition circuit (W) 34. The weight addition circuit 34 multiplies the predicted image data by the weight W and outputs the weight predicted image data to the arithmetic circuit 33.
[0029]
The output data (image data) of the arithmetic circuit 10 is input to the frame memory group 11 and the image enlargement circuit 31 as described above. The image enlargement circuit 31 enlarges the image data generated by the arithmetic circuit 10 so as to have the same size as the image size of the upper layer, and outputs it to the weight addition circuit (1-W) 32. The weight addition circuit 32 multiplies the output data of the image enlargement circuit 31 by the weight (1-W) and outputs the result to the arithmetic circuit 33 as weight prediction image data.
[0030]
The arithmetic circuit 33 adds the output data of the weight addition circuit 32 and the output data of the weight addition circuit 34 and outputs the result to the arithmetic circuit 53 as predicted image data. The output data of the arithmetic circuit 33 is also input to the arithmetic circuit 60, added to the output data of the inverse DCT circuit 59, input to the frame memory group 61, and then the predicted reference data frame of the image data to be encoded. Used as The arithmetic circuit 53 calculates a difference between the image data to be encoded and the output data (predicted image data) of the arithmetic circuit 33 and outputs the difference as difference data. However, in the case of an intra-frame encoded macroblock, the arithmetic circuit 53 outputs the image data to be encoded to the DCT circuit 54 as it is.
[0031]
The DCT circuit 54 performs DCT (discrete cosine transform) processing on the output data of the arithmetic circuit 53, generates DCT coefficients, and outputs the DCT coefficients to the quantization circuit 55. As in the case of MP @ ML, the quantization circuit 55 quantizes the DCT coefficient according to the quantization scale determined from the data accumulation amount of the transmission buffer 57 and outputs the quantized data to the variable length coding circuit 56. The variable length coding circuit 56 performs variable length coding on the quantized data (quantized DCT coefficient), and then outputs it as a bit stream of the upper layer via the transmission buffer 57.
[0032]
The output data of the quantization circuit 55 is also inversely quantized by the inverse quantization circuit 58 at the quantization scale used in the quantization circuit 55. The output data of the inverse quantization circuit 58 (DCT coefficient selected by being inversely quantized) is supplied to the IDCT circuit 59, subjected to inverse DCT processing by the IDCT circuit 59, and then input to the arithmetic circuit 60. In the arithmetic circuit 60, the output data of the arithmetic circuit 33 and the output data (difference data) of the inverse DCT circuit 59 are added, and the output data is input to the frame memory group 61.
[0033]
The variable length coding circuit 56 also receives the motion vector and prediction mode detected by the motion vector detection circuit 52, the quantization scale used by the quantization circuit 55, and the weight W used by the weight addition circuits 34 and 32. Are encoded and supplied to the buffer 57 as encoded data. The encoded data is transmitted as a bit stream via the buffer 57.
[0034]
Next, an example of a spatial scalability decoder will be described with reference to FIG. After the lower layer bit stream is input to the reception buffer 21, it is decoded in the same manner as MP @ ML. However, the output data of the arithmetic circuit 25 is output to the outside, stored in the frame memory group 26, and used as predicted image data of an image signal to be decoded thereafter. After being enlarged to the same image size as that of the image signal, it is also used as the predicted image data of the upper layer.
[0035]
The upper layer bit stream is supplied to the variable length decoding circuit 72 via the reception buffer 71, and the variable length code is decoded. At this time, the quantization scale, the motion vector, the prediction mode, and the weighting coefficient are decoded together with the DCT coefficient. The quantized data decoded by the variable length decoding circuit 72 is inversely quantized by the inverse quantization circuit 73 using the decoded quantization scale, and then DCT coefficients (DCT obtained by inverse quantization). Coefficient) is supplied to the IDCT circuit 74. The DCT coefficient is subjected to inverse DCT processing by the IDCT circuit 74, and output data is supplied to the arithmetic circuit 75.
[0036]
The motion compensation circuit 77 generates predicted image data according to the decoded motion vector and the prediction mode, and inputs the predicted image data to the weight addition circuit 84. The weight addition circuit 84 multiplies the decoded weight W by the output data of the motion compensation circuit 77 and outputs the result to the arithmetic circuit 83.
[0037]
The output data of the arithmetic circuit 25 is output as reproduced image data of the lower layer and also output to the frame memory group 26. At the same time, the image data is enlarged to the same image size as the image size of the upper layer by the image signal enlargement circuit 81. Thereafter, it is output to the weight addition circuit 82. The weight addition circuit 82 multiplies the output data of the image signal expansion circuit 81 by (1-W) using the decoded weight W 1 and outputs the result to the arithmetic circuit 83.
[0038]
The arithmetic circuit 83 adds the output data of the weight addition circuit 84 and the output data of the weight addition circuit 82 and outputs the result to the arithmetic circuit 75. In the arithmetic circuit 75, the output data of the IDCT circuit 74 and the output data of the arithmetic circuit 83 are added and output as a reproduction image of the upper layer, supplied to the frame memory group 76, and then the predicted image of the image data to be decoded Use as data.
[0039]
In the above description, the processing of the luminance signal has been described, but the processing of the color difference signal is similarly performed. However, in this case, the motion vector used is a luminance signal halved in the vertical and horizontal directions.
[0040]
Although the MPEG system has been described above, various other high-efficiency encoding systems for moving images have been standardized. For example, ITU-T prescribes H.261 and H.263 as the main encoding methods for communication. H.261 and H.263 are basically a combination of motion-compensated predictive coding and DCT transform coding as in the MPEG system, and the details of the header information differ, but the image signal coding device ( The encoder and the image signal decoding device (decoder) have the same configuration.
[0041]
Also in the above-described MPEG system, standardization of a new high-efficiency encoding system for moving image signals called MPEG4 is in progress. A major feature of MPEG4 is that it is possible to encode and process an image in units of objects (divide it into a plurality of images). On the decoding side, an image signal of each object, that is, a plurality of image signals is synthesized to reconstruct one image.
[0042]
For example, a chroma key method is used for an image composition system that composes a plurality of images to form one image. This is a method in which a predetermined object is photographed in front of a background of a specific uniform color such as blue, an area other than blue is extracted therefrom, and is synthesized with another image. At this time, a signal indicating the extracted region is called a key signal.
[0043]
Next, a method for encoding a composite image will be described with reference to FIG. Image F1 is the background and image F2 is the foreground. The foreground F2 is an image that is generated by shooting in front of a background of a specific color and extracting an area other than that color. At this time, the signal indicating the extracted region is the key signal K1. The synthesized image F3 is synthesized using these F1, F2, and K1. When this image is encoded, normally, F3 is directly encoded by an encoding method such as MPEG. At this time, information such as the key signal is lost, and it becomes difficult to re-edit and re-synthesize the image such that the foreground F2 remains unchanged and only the background F1 is changed.
[0044]
On the other hand, as shown in FIG. 20, it is also possible to configure the bit stream of the image F3 by separately encoding the images F1, F2 and the key signal K1, and multiplexing the respective bit streams. is there.
[0045]
FIG. 21 shows a method for obtaining the composite image F3 by decoding the configured bitstream as shown in FIG. The bit stream is decomposed into F1, F2 and K1 bitstreams by demultiplexing, and each is decoded to obtain decoded images F1 ′, F2 ′ and a decoded key signal K1 ′. At this time, if F1 ′ and F2 ′ are synthesized in accordance with the key signal K1 ′, a decoded synthesized image F3 ′ can be obtained. In this case, re-editing and resynthesizing can be performed such that the foreground F2 ′ is left as it is, and only the background F1 ′ is changed with the bit stream as it is.
[0046]
As described above, in MPEG4, each image sequence constituting a composite image such as images F1 and F2 is called VO (VideoObject). An image frame at a certain time of VO is called VOP (Video Object Plane). VOP consists of luminance and color difference signals, and key signals. An image frame means one image at a predetermined time, and an image sequence means a set of image frames at different times. That is, each VO is a set of VOPs at different times. Each VO varies in size and position depending on the time. That is, the size and position of VOPs belonging to the same VO are different.
[0047]
The configurations of the encoder and decoder for encoding and decoding in units of objects described above are shown in FIGS. FIG. 22 shows an example of an encoder. The input image signal is first input to the VO configuration circuit 101. The VO configuration circuit 101 divides an input image for each object and outputs an image signal representing each object (VO). Each VO image signal consists of an image signal and a key signal. The image signal output from the VO configuration circuit 101 is output to the VOP configuration circuits 102-0 to 102-n for each VO. For example, the VO 0 image signal and key signal are input to the VOP configuration circuit 102-0, the VO 1 image signal and key signal are input to the VOP configuration circuit 102-1, and so on. The signal and the key signal are input to the VOP configuration circuit 102-n.
[0048]
In the VO component circuit 101, for example, in the case of an image signal generated with a chroma key as shown in FIG. 20, the VO is composed of each image signal and its key signal as it is. For an image that has no key signal or has been lost, image region segmentation is performed, a predetermined region is extracted, a key signal is generated, and is set to VO.
[0049]
The VOP configuration circuits 102-0 to 102-n extract a minimum rectangular portion including an object in the image from each image frame. At this time, however, the number of pixels in the horizontal and vertical directions of the rectangle is a multiple of 16. The VOP configuration circuits 102-0 to 102-n extract the image signals (luminance and color difference signals) and key signals included in the above-described rectangle and output them. It also outputs a flag indicating the size of the VOP (VOP size) and a flag indicating the position of the VOP in absolute coordinates (VOP POS).
[0050]
Output signals from the VOP configuration circuits 102-0 to 102-n are input to the VOP encoding circuits 103-0 to 103-n and encoded. Outputs of the VOP encoding circuits 103-0 to 103-n are input to the multiplexing circuit 104, configured as one bit stream, and output to the outside as a bit stream.
[0051]
FIG. 23 shows an example of a decoder. The multiplexed bit stream is demultiplexed by the demultiplexing circuit 111 and decomposed into bit streams of each VO. The bit stream of each VO is input to the VOP decoding circuits 112-0 to 112-n and decoded. The VOP decoding circuits 112-0 to 112-n decode the VOP image signal, the key signal, the VOP size flag (VOP size), and the VOP absolute position flag (VOP POS). , Input to the image reconstruction circuit 113. The image reconstruction circuit 113 synthesizes an image using each VOP image signal, a key signal, a size flag (VOP size), and a position in absolute coordinates (VOP POS), and outputs a reproduced image. .
[0052]
Next, an example of the VOP encoding circuit 103-0 (the other VOP encoding circuits 103-1 to 103-n are configured similarly) will be described with reference to FIG. The image signal and key signal constituting each VOP are input to the image signal encoding circuit 121 and the key signal encoding circuit 122, respectively. The image signal encoding circuit 121 performs encoding processing by a method such as MPEG or H.263. The key signal encoding circuit 122 performs encoding processing using, for example, DPCM. There is also a method of encoding a difference signal by performing motion compensation using a motion vector detected by the image signal encoding circuit 121 when encoding a key signal. The bit amount generated by the key signal encoding is input to the image signal encoding circuit 121 and controlled to have a predetermined bit rate.
[0053]
The bit stream of the encoded image signal (motion vector and texture information) and the bit stream of the key signal are input to the multiplexing circuit 123, configured as one bit stream, and output via the transmission buffer 124.
[0054]
FIG. 25 illustrates a configuration example of the VOP decoding circuit 112-0 (the other VOP decoding circuits 112-1 to 112-n are configured similarly). First, the bit stream is input to the demultiplexing circuit 131 and is decomposed into a bit stream of an image signal (motion vector and texture information) and a bit stream of a key signal, and the image signal decoding circuit 132 and the key signal decoding circuit 133 , Respectively. In this case, when encoding is performed with motion compensation of the key signal, the motion vector decoded by the image signal decoding circuit 132 is input to the key signal decoding circuit 133 and used for decoding.
[0055]
The method for encoding an image for each VOP has been described above. This method is currently being standardized as MPEG4 in ISO-IEC / JTC1 / SC29 / WG11. However, a method for efficiently encoding each VOP as described above has not been established yet, and a function such as scalability has not been established.
[0056]
Hereinafter, a method for performing scalable coding of an image in units of objects will be described. As described above, the rendering circuit 155 pastes the texture recorded in the texture memory 152 to the polygon regardless of the image format, whether it is a moving image or a still image, and the content thereof. What can be pasted to one polygon is always one texture stored in the memory, and a plurality of textures cannot be pasted to one polygon. In many cases, the image is compressed and transmitted, and after the compressed bit stream is decoded on the terminal side, the image is stored in a predetermined texture pasting memory.
[0057]
In the conventional case, the number of image signals generated by decoding a bit stream is always one. For example, when an MP @ ML bit stream in MPEG is decoded, one image sequence is decoded. In the case of scalability in MPEG2, when a lower-layer bitstream is decoded, a low-quality image is obtained, and when a lower-layer and upper-layer bitstream is decoded, a high-quality image signal is obtained. In either case, one image sequence is decoded.
[0058]
However, in the case of a system such as MPEG4 that encodes an image in object units, the situation is different. That is, one object may be composed of a plurality of bit streams. In such a case, a plurality of images are obtained for each bit stream. Therefore, for example, a texture cannot be pasted on a three-dimensional object described in VRML or the like.
[0059]
As a method for solving this, it is conceivable to assign one VRML node (polygon) to one image object (VO). For example, in the case of FIG. 21, it can be considered that the background F1 ′ is assigned to one node, and the foreground F2 ′ and the key signal K1 ′ are assigned to one node. However, if one image object consists of multiple bitstreams and multiple images are generated during decoding, the following problems But is there. This will be described with reference to FIGS.
[0060]
A description will be given taking a three-layer scalable coding as an example. In the case of three-layer scalable coding, in addition to the lower layer (base layer), there are two upper layers, namely, a first upper layer (enhancement layer 1, hereinafter referred to as upper layer 1 as appropriate) and a second upper layer ( There is an enhancement layer 2 (hereinafter referred to as upper layer 2 as appropriate). Compared with the image decoded up to the first upper layer, the image decoded up to the second upper layer has improved image quality. Here, the improvement in image quality is the spatial resolution in the case of spatial scalable coding, the frame rate in the case of temporal scalable coding, and the image quality in the case of SNR (Single to Noise Ratio) scalable coding. SNR.
[0061]
In the case of MPEG4 encoded in object units, the relationship between the first upper layer and the second upper layer is one of the following.
(1) The second upper layer includes all areas of the first upper layer.
(2) The second upper layer corresponds to a partial region of the first upper layer.
(3) The second upper layer corresponds to a wider area than the first upper layer.
[0062]
The relationship (3) exists when scalable coding of three or more layers is performed. This is because the first upper layer corresponds to a partial region of the lower layer and the second upper layer includes all the regions of the lower layer, or the first upper layer is a partial region of the lower layer. Correspondingly, the second upper layer corresponds to a wider area than the first upper layer and corresponds to a partial area of the lower layer. In the case of the relationship (3), when decoding up to the first upper layer, only a part of the lower layer image is improved in image quality, and when decoding up to the second upper layer, all of the larger area or lower layer images The image quality of the area is improved. In the relationship (3), the shape of the VOP may be a rectangle or an arbitrary shape.
[0063]
FIG. 26 to FIG. 31 show examples of spatially scalable coding with three layers. FIG. 26 shows an example in which the VOPs are all rectangular in the spatial scalability in the relationship (1). FIG. 27 shows an example of the spatial scalability in the relationship (2) when the VOP shape is a rectangle. Further, FIG. 28 shows an example of the case where the VOP shapes of all layers are rectangular in the spatial scalability in the relationship (3). FIG. 29 shows the spatial scalability in the relationship (3), where the VOP shape of the first upper layer is an arbitrary shape, and the VOP shapes of the lower layer and the second upper layer are rectangular. An example is shown. FIG. 30 and FIG. 31 show an example of the spatial scalability in the relationship (1), where the VOP has a rectangular shape and an arbitrary shape, respectively.
[0064]
Here, as shown in FIG. 26, when the image quality of the entire image is improved, it is sufficient to display one image with the highest image quality as in the conventional scalable encoding such as MPEG2. However, there are cases as shown in FIG. 27, FIG. 28, and FIG. 29 in MPEG4 encoded in object units. For example, in the case of FIG. 27, when the bitstreams of the lower layer and the upper layers 1 and 2 are decoded, after the resolution conversion of the images of the lower layer and the upper layer 1, the two image sequences after the resolution conversion are converted into the upper layer 2 The entire image is reconstructed by combining with the decoded image sequence. In the case of FIG. 29, only the upper layer 1 and the lower layer may be decoded, and only the image of the upper layer 1 may be output and synthesized with another image sequence decoded from another bitstream.
[0065]
[Problems to be solved by the invention]
As described above, when an image is encoded in units of objects, a method of simply assigning one node to one object generates a texture when a plurality of images are generated for one object. As a result, there is a problem that it cannot be pasted to the object.
[0066]
The present invention has been made in view of such a situation, and even when a plurality of images are generated for one object, the images can be reliably pasted on the object as a texture. Is.
[0067]
[Means for Solving the Problems]
The present invention of The image signal multiplexing apparatus selects a scene descriptor indicating spatial configuration information describing a predetermined object, and selects a bit stream that constitutes the predetermined object from a plurality of hierarchically encoded bit streams. Selecting means for selecting; generating means for generating an object descriptor indicating information relating to an object composed of the selected bitstream; a start code; a selected scene descriptor and bitstream; and an generated object descriptor And multiplexing means for outputting in the order of a start code, a scene descriptor, a predetermined number of object descriptors, and a predetermined number of bitstreams.
[0068]
The present invention of In the image signal multiplexing method, a scene descriptor indicating spatial configuration information describing a predetermined object is selected, and a bit stream constituting the predetermined object is selected from a plurality of hierarchically encoded bit streams. A selection step to select, a generation step to generate an object descriptor indicating information about an object comprising the selected bitstream, a start code, a selected scene descriptor and bitstream, and a generated object descriptor And a multiplexing step of outputting in order of a start code, a scene descriptor, a predetermined number of object descriptors, and a predetermined number of bitstreams.
[0069]
The present invention of The recording medium selects a scene descriptor indicating spatial configuration information describing a predetermined object, and selects a bit stream constituting the predetermined object from a plurality of hierarchically encoded bit streams. A step of generating an object descriptor indicating information about an object comprising the selected bitstream; and multiplexing the start code, the selected scene descriptor and bitstream, and the generated object descriptor , A program for causing a computer to execute processing including a start code, a scene descriptor, a predetermined number of object descriptors, and a multiplexing step for outputting in the order of a predetermined number of bitstreams.
[0078]
The transmission medium according to claim 20 includes spatial configuration information describing an object, bit streams of a plurality of layers having different qualities constituting the object, and dependency information indicating dependency relationships of information between different bit streams. Based on the dependency information, the separation step for separating the information on the spatial configuration information, the bit streams of the plurality of layers constituting the object, and the information on the object from the multiplexed bit stream in which the information on the object including at least is multiplexed and transmitted A spatial configuration information describing a predetermined object, or a control step for controlling the processing in the separation step to select a bit stream of a plurality of layers constituting the object, and an analysis step for analyzing the selected spatial configuration information And multiple levels of bitstreams A decoding step for decoding the output signal, a mixing step for mixing output signals corresponding to the same object among the output signals decoded in the decoding step, and mixing with the output data analyzed in the analysis step based on the information about the object A program including a reconstruction step of reconstructing an image signal from the output signal mixed in the step is transmitted.
[0079]
The present invention of In the image signal multiplexing apparatus and method, and the recording medium program, a scene descriptor indicating spatial configuration information describing a predetermined object is selected, and a plurality of hierarchically encoded bitstreams are selected. A bitstream comprising a given object is selected, an object descriptor is generated indicating information about the object comprising the selected bitstream, a start code, the selected scene descriptor and bitstream are generated, and The object descriptors are multiplexed and output in the order of a start code, a scene descriptor, a predetermined number of object descriptors, and a predetermined number of bitstreams.
[0083]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below. First, the bit stream multiplexer and demultiplexer in the first embodiment will be described with reference to FIG. In the following description, encoded audio and video bit streams (Elementary Stream (ES)) are described as being recorded in advance in a predetermined storage device 202. The bit stream may be directly input from the multiplexing device to the multiplexing circuit 203 without passing through the storage device 202. Hereinafter, the encoding and decoding methods will be described assuming the MPEG4 method, but any method can be applied in the same manner as long as the image is divided into a plurality of images and encoded.
[0084]
The storage device 202 includes a bit stream ES (Elementary Stream) corresponding to each AV (audio and video) object, object stream information OI necessary for decoding each bit stream, and a two-dimensional or three-dimensional scene. A scene descriptor SD (Scene Descriptor) describing (a virtual space constituted by images to be transmitted) is recorded. Here, the object stream information OI includes, for example, a buffer size necessary for decoding, a time stamp of each access unit (frame or VOP), and the like. Details will be described later.
[0085]
In the object information OI, all the information of the bit stream ES corresponding to each AV (audio and video) object is described. The object descriptor generation circuit 204 generates an object descriptor OD (Object Descriptor) corresponding to the object information OI supplied from the storage device 202.
[0086]
The multiplexing circuit 203 multiplexes the bit stream ES and the scene descriptor SD recorded in the storage device 202 and the object descriptor OD supplied from the object descriptor generation circuit 204 in a predetermined order. The bit stream FS is transmitted.
[0087]
Here, the configuration of the bit stream forming each object will be described. For example, a scene as shown in FIG. 21 includes two objects, a background F1 ′ and a foreground F2 ′. However, the key signal K1 ′ and the foreground F2 ′ are composed of one bit stream ES. Therefore, in the case of FIG. 21, when composed of two video objects VO and not using scalable coding, each VO is composed of one bit stream ES.
[0088]
In the case of FIGS. 26 to 29, the frame is composed of one video object VO. However, in this case, since scalable encoding is performed, one video object VO is composed of three bit streams ES. FIG. 26 to FIG. 29 show examples of scalable encoding of three layers, but the number of layers may be arbitrary.
[0089]
In FIGS. 30 and 31, the scene is composed of two video objects VO of the background (FIG. 30) and the foreground (FIG. 31), and each video object VO is composed of three bit streams ES.
[0090]
The user can arbitrarily set which video object is displayed by sending a request signal from the terminal, and which layer is displayed in the case of scalable coding.
[0091]
In the embodiment of FIG. 1, the user transmits a request signal REQ for specifying a necessary video object and a bit stream from an external terminal (not shown) to the transmission side. The request signal REQ is supplied to the stream control circuit 201. Object stream information OI of the bit stream of each video object is recorded in the storage device 202. As described above, the object stream information OI includes, for example, information indicating how many bitstreams a given object is composed of, information necessary for decoding each bitstream, buffer size, and other information for decoding. Contains information such as whether a bitstream is required.
[0092]
The stream control circuit 201 refers to the object stream information OI supplied from the storage device 202 according to the request signal REQ, determines which bit stream to transmit, and multiplexes the stream request signal SREQ into the multiplexing circuit 203 and the storage device. 202 and the object descriptor generation circuit 204. Further, the storage device 202 reads a predetermined bit stream ES and scene descriptor SD in accordance with the stream request signal SREQ and outputs the read bit stream ES and the scene descriptor SD to the multiplexing circuit 203.
[0093]
The object descriptor generation circuit 204 reads the object stream information OI related to the bit stream of each object (VO) recorded in the storage device 202 according to the stream request signal SREQ, and information on the bit stream requested by the stream request signal SREQ. Extract only the object descriptor OD. Also, the object descriptor occurrence time Road 204 generates an ID number OD_ID indicating which object corresponds to the object descriptor OD. For example, in the case of FIG. 26, when only the lower layer and the upper layer 1 are requested for a predetermined object, the object descriptor generation circuit 204 uses only the information of the lower layer and the upper layer 1 as the object stream information OI. And an object descriptor OD is generated, and an ID number OD_ID indicating the object is generated and written to the object descriptor OD. Then, the object descriptor OD generated in this way is supplied to the multiplexing circuit 203. Details of the syntax of the object descriptor OD and the object stream information OI and the scene descriptor SD will be described later.
[0094]
Next, the operation of the multiplexing circuit 203 will be described with reference to FIG. The multiplexing circuit 203 is supplied with bit streams ES1 to ESn to be transmitted in accordance with the stream request signal SREQ. Each bit stream ES1 to ESn is supplied to the switch 231. Similarly, the scene descriptor SD and the object descriptor OD are supplied to the switch 231. Further, the multiplexing circuit 203 is provided with a start code generation circuit 232, and the start code generated by the start code generation circuit 232 is also supplied to the switch 231. The switch 231 outputs data obtained by switching connections in a predetermined order to the outside as a multiplexed bit stream FS.
[0095]
First, the start code generated by the start code generation circuit 232 is output as the multiplexed bit stream FS. Next, the connection of the switch 231 is switched, and the scene descriptor SD is output. After the scene descriptor SD is output, the connection of the switch 231 is switched again, and the object descriptor OD is output. Since there are as many object descriptors OD as there are objects, the same number of object descriptors are output. FIG. 2 shows a case where the number of objects is three. After the object descriptor OD is output, the connection of the switch 231 is switched again, and the bit streams ES1 to ESn are selected and output for each predetermined data size. As shown in FIG. 1, the multiplexed bit stream ES is supplied to the demultiplexing circuit 205 after passing through a predetermined transmission path.
[0096]
Next, details of the demultiplexing circuit 205 will be described with reference to FIG. First, the multiplexed bit stream FS is supplied to the switch 241. First, the switch 241 recognizes each subsequent data by detecting a start code. After detecting the start code, the scene descriptor SD is read from the switch 241 and output. Next, the connection of the switch 241 is changed, and the object descriptor OD is read and output. There are as many object descriptors OD as the number of objects, and they are output sequentially. After all the object descriptors OD are output, the connection of the switch 241 is changed again, and each bit stream ES1 to ESn is read and output according to a predetermined connection.
[0097]
The read scene descriptor SD is supplied to a parsing circuit (parser) 208 and analyzed as shown in FIG. The parsed scene description is supplied to the reconstruction circuit 209 as three-dimensional object information. The three-dimensional object information is actually composed of information such as nodes and polygons, but in the following description, it will be described as a node as appropriate.
[0098]
Further, the read object descriptor OD is supplied to the parsing circuit (parser) 206 and analyzed as shown in FIG. The parsing circuit 206 identifies the type and number of necessary decoders, and causes the necessary decoders 207-1 to 207-n to supply the respective bit streams ES1 to ESn from the demultiplexing circuit 205. Further, the buffer amount necessary for decoding each bit stream is read from the object descriptor OD, and is output from the syntax analysis circuit 206 to each decoder 207-1 to 207-n. Each of the decoders 207-1 to 207-n is initialized based on initialization information such as a buffer size supplied from the syntax analysis circuit 206 (that is, transmitted by the object descriptor OD). Further, the syntax analysis circuit 206 reads the ID number OD_ID of each object descriptor OD in order to identify which object each bitstream ES1 to ESn belongs to. Then, the ID number OD_ID of each object descriptor OD is output from the syntax analysis circuit 206 to the decoders 207-1 to 207-n that decode the bit stream described in the object descriptor OD.
[0099]
Each of the decoders 207-1 to 207-n decodes the bit stream based on a predetermined decoding method corresponding to encoding, and outputs a video or audio signal to the reconstruction circuit 209. Each decoder 207-1 to 207-n outputs an ID number OD_ID indicating to which object the image belongs to the reconstruction circuit 209. In the case of an image signal, each of the decoders 207-1 to 207-n decodes a signal (POS, SZ) indicating the position and size from the bit stream and outputs the decoded signal to the reconstruction circuit 209. Further, in the case of an image signal, the decoders 207-1 to 207-n decode a signal indicating a transparency (key signal) from the bit stream and output the decoded signal to the reconstruction circuit 209.
[0100]
Next, the correspondence between signals for reconstructing an image and the reconstruction circuit 209 will be described with reference to FIGS. 4 and 5. FIG. FIG. 4 shows an example in the case where scalable coding is not performed, and FIG. 5 shows an example in the case where scalable coding is performed.
[0101]
In FIG. 4, the reconstruction circuit 209 includes a synthesis circuit 252, and an image signal generated by the synthesis circuit 252 is supplied to the display 251 and displayed. In FIG. 4, the synthesis circuit 252 and the display 251 are shown as the reconstruction circuit 209, but this is to show how the image constituted by the synthesis circuit 252 is shown on the display 251. In practice, the reconstruction circuit 209 does not include a display.
[0102]
In FIG. 4, a rectangular image sequence and a triangular pyramid generated by CG are displayed on the screen of the display 251. The decrypted texture is also attached to the object of the triangular pyramid. Here, the texture may be a moving image or a still image.
[0103]
FIG. 4 shows the correspondence between the scene descriptor SD and the output screen. As the scene descriptor SD, for example, a descriptor such as VRML is used. The scene descriptor SD is composed of a group of descriptions called nodes. There is a parent (root) node SD0 describing how to arrange each object in the entire image. As a child node, there is a node SD1 describing information about a triangular pyramid. In addition, information regarding a rectangular plane on which an image is pasted is described in a node SD2 as a child node of the root node SD0. In the example of FIG. 4, the image signal is composed of three video objects VO. Information relating to the background as the first video object VO is described in the node SD2. In addition, information regarding a plane for attaching the sun as the second video object VO is described in the node SD3. Further, information regarding a plane on which a person as the third video object VO is pasted is described in the node SD4. Nodes SD3 and SD4 are child nodes of node SD2.
[0104]
Therefore, one scene descriptor SD is constituted by the nodes SD0 to SD4. Each node SD0 to SD4 corresponds to one three-dimensional or two-dimensional object. In the example of FIG. 4, the node SD0 is an object of the entire scene, the node SD1 is a triangular pyramid object, the node SD2 is a background object, the node SD3 is a sun object, and the node SD4 is a person object. It corresponds. When a texture is pasted to each node, a flag indicating which bit stream corresponds to each node is required. In order to identify this, the ID number OD_ID of the object descriptor supplied from the decoder of the corresponding bit stream is described in each node. As a result, one object descriptor OD corresponds to one node. Thus, one video object VO is pasted on one two-dimensional or three-dimensional object.
[0105]
Each node SD0 to SD4 constituting the scene descriptor SD is interpreted by the syntax analysis circuit 208 and supplied to the synthesis circuit 252 of the reconstruction circuit 209 as three-dimensional object information. Also, the decoders 207-1 to 207-4 are supplied with the bitstreams ES1 to ES4 from the demultiplexing circuit 205, and are supplied with the ID number OD_ID of the corresponding object descriptor OD from the syntax analysis circuit 206. . Each decoder 207-1 to 207-4 decodes the bit stream, and in addition to an ID number OD_ID and a decoded signal (image or audio), in the case of an image, a key signal and a signal (POS) indicating the position and size of the image , SZ) is supplied to the synthesis circuit 252 of the reconstruction circuit 209 as a decoded signal. Here, the position of the image means a relative position with respect to the parent node one level above that node belongs.
[0106]
A configuration example of the synthesis circuit 252 is shown in FIG. In FIG. 6, parts corresponding to those shown in FIG. 14 are given the same reference numerals. Input 3D object information (including nodes SD0 to SD4 and polygon information), image signal (Texture), key signal (key signal), ID number OD_ID, position and size signals (POS, SZ) are Are supplied to the object composition circuits 271-1 to 271-n, respectively. One object composition circuit 271-i corresponds to one node SDi (i = 1, 2, 3,..., N). The object synthesis circuit 271-i receives the decoded signal having the ID number OD_ID indicated by the node SDi from the decoder 207-i, and in the case of an image signal, pastes it on the generated two-dimensional or three-dimensional object. As described above, when the ID number OD_ID and the decoded signal are supplied to the corresponding object synthesis circuit 271-i, it is necessary to search which node each decoded signal corresponds to. Therefore, the correlation is recognized by comparing the ID number OD_ID supplied to the reconfiguration circuit 209 with the ID number OD_ID including the node. Based on the recognition result, the decoded signal is supplied to the object composition circuit 271-i to which the corresponding node is supplied.
[0107]
The texture (image signal) to be pasted, the signal indicating the transparency (key signal), and the signal indicating the position and size (VOP, SZ) supplied from the decoder 207-i are predetermined in the memory group 151-i. Is stored in the area. Similarly, the node (two-dimensional or three-dimensional object information) supplied from the syntax analysis circuit 208 is stored in a predetermined storage area of the memory group 151-i. The texture (image signal) is stored in the texture memory 152-i, the signal indicating the transparency (key signal) and the ID number OD_ID are stored in the grayscale memory 153-i, and the node is stored in the three-dimensional information memory 154-i. The An ID number OD_ID is supplied and used to identify the object. Furthermore, the signals (POS, SZ) indicating the position and the magnitude may be stored in any memory, but for example, in this example, are stored in the gray scale memory 153-i. Here, the three-dimensional object information includes polygon formation information and illumination information. A signal indicating the position and size is stored at a predetermined position in the memory group 151-i.
[0108]
The rendering circuit 155-i forms a two-dimensional or three-dimensional object with polygons based on the nodes recorded in the three-dimensional information memory 154-i. The rendering circuit 155-i reads a signal indicating a predetermined texture and transparency from the texture memory 152-i and the grayscale memory 153-i and pastes it on the generated three-dimensional object. The signal indicating the transparency indicates the transparency of the texture at the corresponding position, and indicates the transparency of the object at the position where the texture at the corresponding position is pasted. The rendering circuit 155-i supplies the signal with the texture pasted to the two-dimensional conversion circuit 156. Similarly, a signal indicating the position and size of the image (relative position with respect to the parent node) is read out from a predetermined position (in this example, the grayscale memory 153-i) of the memory group 151-i, The data is output to the two-dimensional conversion circuit 156.
[0109]
The two-dimensional conversion circuit 156 is supplied with two-dimensional or three-dimensional objects to which textures are pasted from the object composition circuits 271-1 to 271-n corresponding to the number of nodes. The two-dimensional conversion circuit 156 maps a three-dimensional object to a two-dimensional plane and converts it into a two-dimensional image signal based on viewpoint information supplied from the outside and signals (POS, SZ) indicating the position and size of the image. To do. The three-dimensional object converted into the two-dimensional image signal is further output to the display 251 and displayed. When all the objects are two-dimensional objects, output data from the rendering circuits 155-1 to 155-n are synthesized according to the transparency (key signal) and a signal indicating the position and size of the image and output. Is done. In this case, conversion based on the viewpoint is not performed.
[0110]
Next, an example when the scalable coding of FIG. 5 is performed will be described. In this case, the reconfiguration circuit 209 includes a mixing circuit 261 and a synthesis circuit 252, and an image signal generated by the mixing circuit 261 and the synthesis circuit 252 is supplied to the display 251 and displayed. In FIG. 5, as in FIG. 4, the mixing circuit 261, the synthesis circuit 252, and the display 251 are shown as the reconstruction circuit 209, but this shows which image is composed of the mixing circuit 261 and the synthesis circuit 252. The display 251 does not include a display in the reconstruction circuit 209 in practice. In the example of FIG. 5, a rectangular image sequence and a triangular pyramid generated by CG are displayed on the display 251. The decrypted texture is also attached to the object of the triangular pyramid. Here, the texture may be a moving image or a still image.
[0111]
FIG. 5 shows the correspondence between the scene descriptor SD and the output screen. In the case of FIG. 5, there is a parent node SD0 describing how to arrange each object in the entire image. As its child nodes, there are a node SD1 in which information about a triangular pyramid is described and a node SD2 in which information about a rectangular plane to which an image is pasted are described. Unlike the example of FIG. 4, the image signal corresponding to the node SD2 in FIG. 5 is composed of one video object VO. However, in the case of FIG. 5, it is assumed that the image corresponding to the node SD2 has been subjected to three-layer scalable coding, and a video object VO is composed of three video object layers. In addition, although FIG. 5 illustrates an example of three layers, the number of layers may be arbitrary.
[0112]
Each node SD0 to SD2 constituting the scene descriptor SD is interpreted by the syntax analysis circuit 208, and the analysis result is supplied to the synthesis circuit 252. Bit streams ES1 to ESn are supplied from the demultiplexing circuit 205 to the decoders 207-1 to 207-4, and the ID number OD_ID of the corresponding object descriptor OD is supplied from the syntax analysis circuit 206. After the decoder 207-1 to 207-4 decodes the bit stream, in addition to the decoded signal, in the case of an image, a key signal, a signal (VOP, SZ) indicating the position and size of the image, and a signal FR indicating the magnification are received. This is supplied to the mixing circuit 261. Here, the position of the image means the relative position of each layer in the same video object VO. Each decoder 207-1 to 207-4 supplies the ID number OD_ID to the synthesis circuit 252. Since the configuration of the synthesis circuit 252 is the same as that shown in FIG. 6, the description thereof is omitted here.
[0113]
As described above, when the ID number OD_ID and the decoded signal are supplied to the corresponding object synthesis circuit 271-i, it is necessary to search which node each decoded signal corresponds to. Therefore, the correlation is recognized by comparing the ID number OD_ID supplied to the reconfiguration circuit 209 with the ID number OD_ID included in the node. Based on the recognition result, the decoded signal is supplied to the object composition circuit 271-i to which the corresponding node is supplied.
[0114]
In the case of scalable coding, the bit stream of each layer (VOL) belongs to the same video object VO and therefore has the same ID number OD_ID. One video object VO corresponds to one node, and one texture memory 152-i corresponds to one video object VO corresponding to the video object VO. Therefore, in the case of scalable coding, the output of each layer (the output of the decoders 207-2 to 207-4) is once supplied to the mixing circuit 261 and synthesized into one image sequence.
[0115]
The mixing circuit 261 first synthesizes the image of each layer based on the image signal, the key signal, the signal indicating the magnification, and the signal indicating the position and size of the image supplied from each decoder 207-2 to 207-4. Thereafter, the data is output to the synthesis circuit 252. Therefore, the composition circuit 252 can correspond one image sequence to one object.
[0116]
For example, when scalable coding as shown in FIG. 29 is performed and the lower layer and the upper layer 1 are transmitted and decoded, the image signal of the lower layer is subjected to resolution conversion based on the signal FR indicating the magnification. . Next, the decoded image of the upper layer 1 is combined with this image at a position corresponding to the decoded image according to the key signal.
[0117]
The image sequence synthesized by the mixing circuit 261 is supplied to the synthesis circuit 252. The composition circuit 252 constructs an image in the same manner as in FIG. 4 and outputs it to the display 251 to obtain a final output image.
[0118]
As described above, in this example, one object (video object VO in the case of video) is allocated to one node, and the mixing circuit is provided in the previous stage of the memory group 151 that stores the texture, three-dimensional information, and the like in the rendering circuit 155. 261 is provided, and a plurality of images are mixed according to a predetermined key signal and then recorded in the texture memory 152 to enable texture mapping of an image signal having a plurality of resolutions.
[0119]
As described above, in the example of FIG. 1, for a certain object, a descriptor that records the system information of the bit stream that constitutes the object is generated, and information on the bit stream that must be decoded at that time is generated. Only the bit stream described in the descriptor is decoded, so that a combination of the decodable bit streams can be identified and a predetermined signal can be decoded. In this case, the descriptor is generated and transmitted one-to-one on the transmission side and the reception side.
[0120]
Next, FIGS. 7 to 9 show the structure of the object descriptor OD. FIG. 7 shows the overall configuration (syntax) of the object descriptor OD.
[0121]
Node ID is a 10-bit flag indicating the ID number of the descriptor. Corresponds to the above OD_ID. StreamCount is an 8-bit flag indicating the number of bit streams ES included in the object descriptor. This number of information, ES_Descriptor, necessary for decoding the bitstream ES is transmitted. Furthermore, extensionFlag is a flag indicating whether or not to transmit other descriptors. When this value is 1, other descriptors are transmitted.
[0122]
ES_Descriptor is a descriptor indicating information on each bit stream. FIG. 8 shows the configuration (syntax) of ES_Descriptor. ES_Number is a 5-bit flag indicating an ID number for identifying the bit stream. StreamType is an 8-bit flag indicating the format of the bit stream, for example, MPEG2 video. Furthermore, QoS_Descriptor is an 8-bit flag indicating a request to the network at the time of transmission.
[0123]
ESConfigParams is a descriptor in which information necessary for decoding the bitstream is described, and its configuration (syntax) is shown in FIG. Details of ESConfigParam are described in MPEG4 System VM.
[0124]
FIG. 10 shows a scene descriptor for pasting a moving image. SFObjectID is a flag indicating an ID number OD_ID that is an ID of an object descriptor of a texture to be pasted. FIG. 11 shows a scene descriptor for pasting a still image. SFObjectID is a flag indicating the ID number OD_ID of the object descriptor of the texture to be pasted. 10 and 11 conform to the VRML node description.
[0125]
Next, FIG. 12 shows a bitstream multiplexing apparatus and a demultiplexing apparatus in the second embodiment. In this embodiment, all bit streams belonging to an object are multiplexed and transmitted. In the first embodiment of FIG. 1, only the bit stream requested from the receiving side is multiplexed and transmitted. At that time, the object descriptor OD was generated according to the bit stream to be transmitted. Since all the bitstreams described in the object descriptor OD are decoded on the receiving side, there is no need to particularly transmit information dependency between the bitstreams.
[0126]
On the other hand, in the second embodiment, the object descriptor OD is stored in the storage device 202 in advance, and all the bit streams recorded in the object descriptor OD are multiplexed and transmitted on the transmission side. To do. At this time, the object descriptor OD in the second embodiment is different from the first embodiment in that the dependency relationship of information between bit streams is described. The other points are the same as those in the first embodiment.
[0127]
The multiplexing circuit 203 reads the scene descriptor SD, the object descriptor OD, and the bit stream group ES recorded in the storage device 202, multiplexes them in a predetermined order, and transmits them. The transmission order and the configuration of the multiplexing circuit 203 are the same as those in the first embodiment. The multiplexed bit stream FS is supplied to the demultiplexing circuit 205 via the transmission path.
[0128]
The user inputs a request signal REQ indicating which object is to be displayed from the terminal. The request signal REQ is supplied to the demultiplexing circuit 205, the syntax analysis circuit 206, and the reconfiguration circuit 209. The parsing circuit 206 analyzes each transmitted object descriptor OD, generates a signal SREQ requesting a necessary bit stream, and supplies the signal SREQ to the demultiplexing circuit 205. When the user requests a predetermined bitstream, the other bitstreams necessary for decoding it and which bitstream is required are recorded in the object descriptor OD.
[0129]
The demultiplexing circuit 205 supplies only the necessary bit stream to the decoders 207-1 to 207-n according to the request signal REQ from the user and the signal SREQ for requesting the necessary bit stream, and the necessary object descriptor OD. Is supplied to the syntax analysis circuit 206. The parsing circuit 206 analyzes the object descriptor OD, and based on the object descriptor OD and the request signal REQ from the user, the initialization information of the decoders 207-1 to 207-n and the ID number OD_ID are assigned to each decoder 207-1. To 207-n. Thereafter, decoding, synthesis, and display are performed in the same manner as in the first embodiment.
[0130]
As described above, in this example, a descriptor (object descriptor) in which system information of a bit stream constituting the object is recorded is generated for an object, and at that time, it is necessary to decode each bit stream. By recording a flag indicating a bitstream and decoding a predetermined bitstream according to the flag described in the descriptor, it is possible to identify a combination of decodable bitstreams and decode a predetermined signal To do. In this case, after the descriptor is generated once on the transmission side, a common descriptor is transmitted to all recipients.
[0131]
In the second embodiment, unlike the first embodiment, the object descriptor OD describes information for identifying another bit stream necessary for decoding a predetermined bit stream. . The object descriptor OD in the second embodiment will be described. The overall configuration of the object descriptor OD is the same as that in the first embodiment shown in FIG.
[0132]
FIG. 13 shows an ES_Descriptor that describes information about each bitstream. isOtherStream is a 1-bit flag that indicates whether another bitstream is required to decode this bitstream. If this value is 0, this bitstream can be decoded alone. When the value of isOtherStream is 1, this bit stream cannot be decoded alone.
[0133]
streamCount is a 5-bit flag indicating how many other bit streams are required. Based on streamCount, ES_Number is transmitted by that number.
[0134]
ES_Number is an ID for identifying a bit stream necessary for decoding. The other configuration of ES_Descriptor is the same as that of the first embodiment. Further, the configuration of ESConfigParams representing information necessary for decoding each bitstream is the same as that in the first embodiment shown in FIG.
[0135]
The above-described processing (multiplexing and demultiplexing) can be realized by a program, and the program can be transmitted (provided) to a user. As a transmission medium, a magnetic disk, CD-ROM, solid In addition to recording media such as memory, communication media such as networks and satellites can be used. Needless to say, the above-described processing can be realized not only as a program but also as hardware.
[0136]
Various modifications and application examples can be considered without departing from the gist of the present invention. Therefore, the gist of the present invention is not limited to the embodiment.
[0137]
【The invention's effect】
The present invention of First Image signal multiplexer And method, and recording medium program According to Duplicate It is possible to texture-map a scalable bitstream in units of objects having several layers.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of an image signal multiplexing device and an image signal demultiplexing device according to the present invention.
FIG. 2 is a block diagram illustrating a configuration example of a multiplexing circuit 203 in FIG.
3 is a block diagram showing a configuration example of a demultiplexing circuit 205 in FIG. 1. FIG.
4 is a diagram illustrating a correspondence relationship between signals for reconstructing an image and the reconstruction circuit 209 in FIG. 1;
FIG. 5 is a diagram illustrating a correspondence relationship between signals for reconstructing an image and the reconstruction circuit 209 in FIG. 1;
6 is a block diagram illustrating a configuration example of a synthesis circuit 252 in FIG. 5;
FIG. 7 is a diagram illustrating a configuration of an object descriptor.
FIG. 8 is a diagram illustrating a configuration of an ES_Descriptor.
FIG. 9 is a diagram showing a configuration of ESConfigParams.
FIG. 10 is a diagram illustrating a configuration of a scene descriptor for moving images.
FIG. 11 is a diagram illustrating a configuration of a scene descriptor for a still image.
FIG. 12 is a block diagram illustrating another configuration example of the image signal multiplexing device and the image signal demultiplexing device according to the present invention.
FIG. 13 is a diagram illustrating a configuration of an ES_Descriptor.
FIG. 14 is a block diagram illustrating a configuration example of a conventional object composition circuit.
FIG. 15 is a block diagram illustrating a configuration example of a conventional image signal encoding device.
FIG. 16 is a block diagram illustrating a configuration example of a conventional image signal decoding apparatus.
FIG. 17 is a block diagram illustrating another configuration example of a conventional image signal encoding device.
FIG. 18 is a block diagram illustrating another configuration example of a conventional image signal decoding device.
FIG. 19 is a diagram for explaining conventional image composition;
FIG. 20 is a diagram for explaining image composition;
FIG. 21 is a diagram for explaining image composition;
FIG. 22 is a block diagram showing still another configuration example of a conventional image signal encoding device.
FIG. 23 is a block diagram illustrating still another configuration example of a conventional image signal decoding device.
24 is a block diagram illustrating a configuration example of the VOP encoding circuit 103-0 in FIG.
25 is a block diagram illustrating a configuration example of the VOP decoding circuit 112-0 in FIG.
FIG. 26 is a diagram illustrating an image object.
FIG. 27 is a diagram illustrating an image object.
FIG. 28 is a diagram illustrating an image object.
FIG. 29 is a diagram illustrating an image object.
FIG. 30 is a diagram illustrating an image object.
FIG. 31 is a diagram illustrating an image object.
[Explanation of symbols]
201 stream control circuit, 202 storage device, 203 multiplexing circuit, 204 object descriptor generation circuit, 205 demultiplexing circuit, 206 syntax analysis circuit, 207-1 to 207-n decoder, 208 syntax analysis circuit, 209 reconfiguration circuit

Claims (4)

Selecting means for selecting a scene descriptor indicating spatial configuration information describing a predetermined object, and selecting the bitstream constituting the predetermined object from a plurality of hierarchically encoded bitstreams; ,
Generating means for generating an object descriptor indicating information on the object composed of the selected bitstream;
The start code, the selected scene descriptor and bitstream, and the generated object descriptor are multiplexed, and the start code, the scene descriptor, the predetermined number of the object descriptors, the predetermined number of the bitstreams are multiplexed. An image signal multiplexing apparatus comprising: multiplexing means for outputting in order. The object descriptor includes at least one of a flag indicating spatial configuration information describing the object, a flag indicating the number of bit streams, and information necessary for decoding the bit stream. 2. An image signal multiplexing device according to 1. Selecting a scene descriptor indicating spatial configuration information describing a predetermined object, and selecting the bit stream constituting the predetermined object from a plurality of hierarchically encoded bit streams; ,
Generating an object descriptor indicating information about the object composed of the selected bitstream;
The start code, the selected scene descriptor and bitstream, and the generated object descriptor are multiplexed, and the start code, the scene descriptor, the predetermined number of the object descriptors, the predetermined number of the bitstreams are multiplexed. And a multiplexing step for outputting in order. Selecting a scene descriptor indicating spatial configuration information describing a predetermined object, and selecting the bit stream constituting the predetermined object from a plurality of hierarchically encoded bit streams; ,
Generating an object descriptor indicating information about the object composed of the selected bitstream;
The start code, the selected scene descriptor and bitstream, and the generated object descriptor are multiplexed, and the start code, the scene descriptor, the predetermined number of the object descriptors, the predetermined number of the bitstreams are multiplexed. A recording medium on which is recorded a program that causes a computer to execute processing including multiplexing steps that are output in order .

Images