JPH104539A - Data multiplexing method and multiplexed data reproduction device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate multiplex data with ease of edit by multiplexing a plurality of objects for each layer in sets of the objects and multiplexing again the object data of each layer so as to simplify the configuration. SOLUTION: Image data and audio data of a base layer of a background, a shell and wheels are inputted a multiplexer 11, in which the data are multiplexed. Image data of a higher layer of a background, a shell and wheels are given to a multiplexer 12, in which the data are multiplexed. Outputs of the multiplexers 11, 12 are inputed to a multiplexer 13, where the data are multiplexed. Thus, multiplexing is applied to object data of each layer of scalability to obtain object data and multiplexed object data for each layer are further multiplexed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data multiplexing method for multiplexing and transmitting digital images, digital audio, digital data, and the like, and a multiplexed data reproducing apparatus for reproducing data.

[0002]

2. Description of the Related Art Conventional digital image, digital audio, and digital data encoding and multiplexing methods are MPEG2 (ISO / IEC JTC1 / SC29 N8) which is an international standard.
01, "ISO / IEC CD 13818-1: Information technology-G
eneric coding of moving pictures and associated au
dio ", 1994.11).

[0003] In the video signal encoding according to the MPEG2 standard, scalability, which is hierarchical encoding, is defined in order to realize simultaneous transmission of codes to decoders having different performances and prioritization when codes are discarded during transmission. . MPEG
The two standards define four scalabilities: spatial scalability, temporal scalability, SNR scalability, and hybrid scalability.

FIG. 8 is an explanatory diagram of spatial scalability in MPEG2 video signal encoding. In FIG. 8, reference numerals 81, 82, and 83 denote high spatial scalability layer images, reference numerals 84, 85, and 86 denote upsampling images of the basic spatial scalability layer, and reference numerals 87, 88, and 89 denote high spatial scalability high layer images. The base layer is low spatial resolution image data, and prediction of an image for compression is performed only between the base layers. Therefore, a low spatial resolution image of the base layer can be decoded by the base layer alone.

[0005] The higher layer is image data having a high spatial resolution, and both the base layer and the higher layer are used for decoding. That is, high spatial resolution image data is decoded by prediction from image information obtained by upsampling the base layer and a restored image of an adjacent higher layer. Using the image data as described above, simultaneous transmission of codes to two decoders, one for decoding a high spatial resolution image and the other for decoding a low spatial resolution image, and a code at the time of transmission and decoding Can be prioritized.

[0006] In the MPEG2 standard video coding, in addition to spatial scalability, a basic layer image of a low temporal resolution image and a high layer image of a high temporal resolution image are considered, and the high layer image is decoded based on difference information from the basic layer. Scalability, low quantization S with large quantization step
Considering a base layer image that is an NR image and a high layer image that is a high SNR image with a small quantization step, the high layer image is SNR scalable to perform decoding based on difference information from the base layer, furthermore, spatial scalability and temporal scalability. , SNR scalability is defined.

FIGS. 9 and 10 are explanatory diagrams of an MPEG2 data multiplexing method, and FIG. 11 is a block diagram of a multiplexed data reproducing apparatus for decoding data multiplexed by the MPEG2 data multiplexing method. In FIG. 9, (a) is a TS packet, (b)
Is a TS packet configuration, (c) is a TS packet header configuration, (d) is a PAT, and (e) is a PMT. next,
In FIG. 10, (a) shows the structure of a PES packet,
(B) shows the configuration of the PES packet header. Further, in FIG. 11, reference numeral 111 denotes a separation unit,
It comprises 1112 CPUs. Reference numeral 112 denotes a synchronization control unit, 113 denotes an image decoder, and 114 denotes an audio decoder. FIG. 12 is a flowchart showing the operation of the multiplexed data reproducing apparatus for MPEG2 multiplexed data.

The conventional MPEG2 constructed as described above
The operation of the data multiplexing method and the multiplexed data reproducing device will be described below.

The image data and audio data are compressed and encoded for each fixed number of samples such as a frame for an image and 1024 for an audio (this is also called a frame in the case of audio in MPEG). The frames are put together to form a packet called a PES packet. FIG. 12 shows the outline of the format of the PES packet. A header is attached to the PES packet,
In the header, a stream ID indicating whether a subsequent data area is image data, audio data, or other data, or DTS (Decoding Time Stamp, DTS) which is time information for synchronously reproducing an image and audio. Decoding time information), PTS (presentation time stamp, presentation time information), and the like. The PES packet is divided into a plurality of 188-byte TS packets to be described later and transmitted.

FIG. 9 schematically shows the format of a TS packet. Each TS packet has a packet-specific number called PID. The PID is not different for each TS packet, but the same PES packet has the same PID. The TS packet sends data following the header as shown in FIG. Fig. 9 shows the PID of the TS packet.
It is sent as part of the header as in (c). In the TS packet area, in addition to the PES packet shown in FIG. 10, it is possible to send program selection information called PSI. FIG.
FIG. 9D shows the structure of the program association table PAT, which is one of the PSIs, and FIG. 9E shows the structure of the program map table PMT. PAT
Is assigned PID = 0, the program number and the PI of the PMT describing the contents of the program number.
D is described.

The PMT has a program number indicating the number of a program to be transmitted, and PIDs of image data PES, audio data PES, and data PES packets.
Multiplex. Further, in order to indicate the above-described scalability relationship, information called a hierarchical descriptor is added to the image PID in the PMT, and information on the scalability and layer to which the image corresponding to each PID belongs can be indicated.

FIG. 11 shows an example of a conventional multiplexed data reproducing apparatus for receiving the multiplexed data and reproducing the data. The received multiplexed data is stored in the buffer 1112 first. The CPU 1111 operates based on the flowchart shown in FIG. 12, and first searches for a PAT of PID = 0 from data transmitted first. Then, the PMT of the PMT of the desired program number set externally from the PAT
Look for ID. Next, a packet having the PID of the PMT obtained from the PAT is extracted from the multiplexed data. PMT
Is extracted, image data PES, audio data PES, and data PES packets are extracted from the PMT.

Finally, the PID of the TS packet in the received data is checked, and the TS necessary for reproducing the program is read.
Extract the packet. The PID is described as PAT1 type, and the program number and PES P
The reason for using two types of tables, PAT and PMT, without describing the ID is that when a large number of programs are multiplexed, the table is prevented from becoming too long. There are things that can be flexibly handled just by changing the content.

[0014] Of the TS packets, image and audio packets are sent to respective decoders 113 and 114.
The synchronization control unit 112 determines the PT multiplexed in the PES packet.
It extracts time information such as S and DTS and controls the timing of synchronous playback of the decoder. The decoders 113 and 114 reproduce and display images and audio synchronously at the time indicated by the synchronization control unit.

[0015]

However, the above configuration has the following problems.

This is a method for multiplexing codes having scalability in object coding.

In image coding, an object coding method for separately coding each component constituting an image, that is, a so-called object such as a background, a person, a moving object, etc., has attracted attention. In object encoding, since encoding is performed for each object, editing such as replacing or removing a specific object or the like and searching for a certain object can be easily performed.

However, in the conventional multiplexing method, there is only a concept of a frame, so that there is a disadvantage that it cannot be handled in an object unit. Extending the conventional technology, a method is considered in which an image is regarded as a set of objects, a PES packet is configured for each object, another PID is allocated for each object, and a relationship between image objects is described in a table such as PAT and PMT. Conceivable. In the above method, when the image has scalability, the image information of the base layer and the image information of the higher layer of a certain component of the image are regarded as objects, respectively, PIDs are respectively assigned, the objects of the base layer and the objects of the higher layer Is described in a table as having a scalability relationship, there is a problem that the description of the relationship between objects becomes complicated, and the structures of the multiplexing device and the multiplexed data reproducing device become complicated.

In so-called object coding,
Incorporating the object-oriented concept, a code that is data and a decoding method that is a method for the code are considered as a set, and a method that inherits only the difference of the decoding method from a parent object to a child object is considered. However, in the case of an image having scalability, when an object having a parent-child relationship is defined for each component, there is a disadvantage that a difference in a decoding method must be described for each component.

[0020]

A multiplexing method according to the present invention comprises:
For a set of scalable object data consisting of a plurality of N layers, the i-th layer (i =
1. . N), the image object data, audio object data, and digital object data corresponding to the i-th object data (i = 1.
N), and N object data from the first object data to the Nth object data are further multiplexed to obtain object data.

According to the multiplexing method of the present invention, the configuration described above eliminates the need to describe the scalability relationship for each image object data corresponding to a plurality of components of an image, thereby simplifying the multiplexing device and the multiplexed data reproducing device. This makes it possible to easily configure, for each scalability hierarchy, and for each object corresponding to a component such as a certain image.

[0022]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a data multiplexing method and a multiplexed data reproducing apparatus according to a first embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram of a multiplexing device according to the first embodiment of the present invention. In FIG. 1, 11 is the first
Selector 111, buffer 11
2, 113, 114 and 115, and a CPU 116.
Reference numeral 12 denotes a second multiplexer, which includes a selector 121, buffers 122, 123, and 124, and a CPU 125. 13
Is a third multiplexer, the selector 131 and the buffer 13
2, 133 and the CPU 134.

FIG. 5 shows a multiplexed data reproducing apparatus according to the first embodiment of the present invention. In FIG. 5, reference numeral 51 denotes a multiplexed data storage medium, 52 denotes a multiplexed data transmission medium, 53 denotes a data demultiplexer, which comprises a CPU 531 and a buffer 532, 54 denotes a controller, and 54 denotes a buffer.
1, a CPU 542, 55 is an image decoder, 56
Is an audio decoder.

The operation of the multiplexing apparatus and the multiplexed data reproducing apparatus according to the present embodiment having the above-described configuration will be described below.

As an original signal, a scalable object code is considered. The object code is described, for example, in "ISO / IEC JTC1 / SC29 / WG11 N1246".

In object encoding, an image is
Encoding is performed separately for each component constituting an image, that is, for a so-called object such as a background, a person, or a moving object.

In this embodiment, a signal as shown in FIG. 2 is considered as an original signal. FIG. 2 is an explanatory diagram of an original signal according to the first embodiment of the present invention.
Reference numerals 4 and 24 are image objects, and 26 is an audio object.

FIG. 2 shows one image in a series of video data accompanied by audio. In FIG.
One image 21 includes a background 22, an object 23 moving in the background, and an audio 26 accompanying the image. Further, the object 23 includes a body 24 and wheels 25. That is, in the object code, the original image 2
1, the background 22, the body 24, the wheels 25 and the audio 26
And compression-code each. In this embodiment, scalability is considered, and it is assumed that the objects of the background 22, the body 24, and the wheels 25 have image object data of the base layer and the higher layer.

This embodiment in which the above-described signal is treated as an original signal will be described with reference to FIGS.

FIG. 3 is an explanatory diagram of a data multiplexing method according to the first embodiment of the present invention. In FIG. 3, 31
Denotes object data of the base layer, and includes image object data 311, 312, 313, and audio data 3.
It consists of 14. Reference numeral 32 denotes object data of a higher layer, which is composed of image object data 321, 322, and 323.

As shown in FIG. 3, in the data multiplexing method of the present invention, the first multiplexing is performed for each object data of each scalability layer to obtain object data. That is, regarding the layer of the basic layer, the background 311, the body 312, the wheels 313, the audio data 3
14 are multiplexed, and for a higher layer, a background 321,
The body 322 and the wheels 323 are multiplexed.

Next, a second multiplexing for further multiplexing the multiplexed object data for each layer is performed.

A data multiplexing apparatus for realizing the above data multiplexing method will be described below with reference to FIG.

The first multiplexing device 11 has a background 311,
The image data and audio 314 of the base layer of the body 312 and the wheels 313 are input and multiplexed. Each data is input to buffers 112, 113, 114 and 115, each of which is added with a PID by the CPU 116, packetized, and output to the selector 111. Selector 1
Reference numerals 11 denote buffers 112, 1 under the control of the CPU 116.
13, 114, and 115 are multiplexed, and the signal information (information of image or audio, scalability information) of the packet corresponding to each PID and the configuration information between image / audio objects (the object Link information indicating the configuration) is output.

The second multiplexer 12 has a background 321,
Image data of the upper layer of the body 322 and the wheels 323 is input and multiplexed. Each data is stored in a buffer 122,
The PIDs are input to 123 and 124, added with a PID by the CPU 125, packetized, and output to the selector 121. The selector 121 multiplexes each packet from the buffers 122, 123, and 124 under the control of the CPU 125, and outputs signal information (information of image or audio, information of scalability) of the packet corresponding to each PID and information of the image / audio object. It outputs configuration information (link information indicating the configuration between objects in FIG. 2).

The output of the first multiplexer and the output of the second multiplexer are input to the third multiplexer 13 and multiplexed. For the data input to the buffers 132 and 133, the same PID and a different SubPID are added by the CPU 134 to the base layer packet and the higher layer packet of the same image component object. In FIG. 3, for example, background objects 311, 3
The packets for 21 are given the same PID. Also, for the same scalability, the same S
ubPID is added. That is, in this embodiment, the objects of the base layer are all the same
The same SubPID is assigned to all objects of the PID and the higher layer.

The selector 131 multiplexes each packet from the buffers 132 and 133 under the control of the CPU 134 and outputs the multiplexed packet together with the PAT, the inter-object information table, and the scalability information table.

FIG. 4 is an explanatory diagram of a multiplexed data structure.
(a) is a TS packet, (b) is a PAT, (c)
Is the inter-object information table (PMT), (d)
Is a scalability information table (PMT).

The multiplexed data includes a PID, which is an identifier of an image component, in each packet as shown in FIG.
It is composed of a TS packet to which SubPID which is a scalability identifier is added.

In the PAT, an inter-object information table corresponding to a program map table of each program,
Two PIDs of the scalability information table are shown.

In the inter-object information table,
The PID of each object and the configuration information (FIG. 2) of the object corresponding to each PID are stored in the scalability information table.
Scalability information corresponding to the PID is shown.

Next, a multiplexed data reproducing apparatus for reproducing multiplexed data configured as described above will be described with reference to FIGS.
This will be described with reference to FIG. The multiplexed data output by the multiplexing device of FIG. 1 is transmitted by the multiplexed data storage medium shown in FIG. 51 or the multiplexed data transmission medium shown in FIG.
The data is input to the multiplex data reproducing device. The multiplexed data is input to the demultiplexer 53, recorded in the buffer 532, and analyzed by the CPU 531 for the PAT and PMT (inter-object information table, scalability information table) according to the flowchart of FIG. The information between the objects is output to the control unit 54, the image data is output to the image decoder 55, and the audio data is output to the audio decoder 56. At this time, the CPU 531 selects and outputs image data corresponding to the performance of the image decoder from the scalability information table and the performance of the image decoder.

The image decoder decodes the image based on the inter-object information, restores the original image, and the audio decoder restores the audio data.

With the above operation, it is not necessary to describe the scalability relationship for each image object data corresponding to a plurality of components of an image, and a multiplexing apparatus and a multiplexed data reproducing apparatus having a simple configuration can be realized. Become.

Further, since the scalability can be determined based on the SubPID, the scalability can be used for the prioritization. When packet discards or errors occur in the transmission path, higher-layer packets can be preferentially discarded, and more error information can be added to the base layer packets, making it easy to achieve graceful degradation etc. become.

Also, with the multiplexed data of the present invention, it is easy to determine the image component and the audio component and the scalability at the packet level.
Editing for each image component, audio component, and scalability becomes easy.

In so-called object encoding,
A method has been considered in which an object-oriented concept is adopted, a code as data and a decoding method as a method for the code are considered as a set, and only a difference between decoding methods is inherited from a parent object to a child object. In the case of an image having scalability, if an object having a parent-child relationship is defined for each component, the difference in the decoding method must be described for each component. However, according to the data multiplexing method of the present invention, the source of the image object is It is sufficient to describe the difference of the decoding method based on scalability only for the image object of the background, which enables efficient method description.

Further, effect control can be easily realized. That is, the scrambler scrambles only the packet corresponding to the image object of the higher layer, and outputs the original signal to the image object of the base layer as it is. By performing the reverse operation in the descrambling device and outputting the result, the effect can be controlled.

FIG. 7 is an explanatory diagram of effect control when spatial scalability is used. In FIG. 7, reference numeral 71 denotes a viewing screen of a legitimate recipient, and 72 denotes a viewing screen of a person other than the legitimate recipient. That is, a legitimate recipient can decode both the higher layer and the base layer, and can restore the legitimate high spatial resolution image 71. On the other hand, non-authorized recipients can restore the image of the base layer only and receive the low spatial resolution image 72 of the base layer. By changing the scalability such as spatial scalability, temporal scalability, and SNR scalability, the resolution, and the like of the image corresponding to the base layer in various ways, it is possible to perform various effect controls according to the application.

Although the present embodiment has been described for the case where the number of scalability layers is two, the present invention can be similarly applied to a case where there are three or more layers.

In this embodiment, the case where scalability exists for an image has been described. However, the present invention is also effective when scalability exists for other data such as voice.

[0053]

As described above, according to the present invention, in a set of objects having scalability of a plurality of layers, a plurality of image objects,
A description of the scalability relationship of each object data corresponding to a plurality of components of images and sounds by multiplexing audio objects and digital data objects into one object, and multiplexing the object data of each layer again. Becomes unnecessary, and a multiplexing device / multiplexed data reproducing device having a simple configuration can be realized, and multiplexed data that can be easily edited for each scalability and each object can be generated.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a multiplexing device according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram of an original signal according to the first embodiment of the present invention.

FIG. 3 is an explanatory diagram of a data multiplexing method according to the first embodiment of the present invention.

4A is a diagram showing a TS packet in multiplexed data. FIG. 4B is a diagram showing a PAT in multiplexed data. FIG. 4C is a diagram showing an inter-object information table (PMT) in multiplexed data. (D) Diagram showing scalability information table (PMT) in multiplexed data

FIG. 5 is a configuration diagram of a multiplexed data reproducing device according to the first embodiment of the present invention.

FIG. 6 is a flowchart of the multiplexed data reproducing apparatus according to the first embodiment of the present invention.

FIG. 7 is an explanatory diagram of effect control when spatial scalability is used.

FIG. 8 is an explanatory diagram of spatial scalability in MPEG2 video signal encoding.

9A is a diagram illustrating a TS packet according to the MPEG2 data multiplexing method. FIG. 9B is a diagram illustrating the configuration of a TS packet according to the MPEG2 data multiplexing method. FIG. 9B is a diagram illustrating the configuration of a TS packet header according to the MPEG2 data multiplexing method. Figure (c) Diagram showing PAT in MPEG2 data multiplexing method (d) Diagram showing PMT in MPEG2 data multiplexing method

FIG. 10 (a) PES by MPEG2 data multiplexing method
Diagram showing the structure of the packet (b) Diagram showing the structure of the PES packet header in the MPEG2 data multiplexing method

FIG. 11 is a configuration diagram of an apparatus for reproducing multiplexed data of MPEG2 data multiplexed data.

FIG. 12 is a flowchart showing the operation of an apparatus for reproducing multiplexed data of MPEG2 multiplexed data;

[Explanation of symbols]

11 first data multiplexer 111 selector 112, 113, 114, 115 buffer 116 CPU 12 second multiplexer 121 selector 122, 123, 124 buffer 125 CPU 13 third multiplexer 131 selector 132, 133 buffer 134 CPU 21 Original image 22, 23, 24, 25 Image object 26 Audio object 31 Object data of base layer 311, 312, 313 Image object data 314 Audio data 32 Object data of higher layer 321, 322, 323 Image object data 51 Multiplex Data storage media 52 Multiplexed data transmission media 53 Data demultiplexer 531 CPU 532 Buffer 54 Control unit 541 Buffer 542 CPU 55 Image Coder 56 Audio decoder 61, 62, 63, 64, 65, 66, 67 Flowchart item 71 Viewing screen of regular receiver 72 Viewing screen of other than normal receiver 81, 82, 83 Spatial scalability higher layer image 84, 85, 86 Space Upsampling images of scalability basic layer 87, 88, 89 Spatial scalability basic layer image 111 Separation unit 1111 Buffer 1112 CPU 112 Synchronization control unit 113 Image decoder 114 Audio decoder 121, 122, 123, 124, 125, 126 Flow chart items

Claims (10)

[Claims] 1. An i-th set of scalable object data sets consisting of a plurality of N layers
The image object data, audio object data, and digital object data corresponding to the layers (i = 1..N) are multiplexed, and the i-th object data (i
= 1. . N) and the first object data
A data multiplexing method characterized by further multiplexing N pieces of object data up to the object data of (1) to obtain object data. 2. An i-th set of scalable object data sets consisting of a plurality of N layers.
The image object data, audio object data, and digital object data corresponding to the layers (i = 1..N) are multiplexed, and the i-th object data (i
= 1. . N), packetize, generate i-th packet data (i = 1... N), and further multiplex N packet data from the first packet data to the N-th packet data to obtain packet data A data multiplexing method, wherein a packet number for identifying a packet is assigned to each of N packets from the first packet data to the N-th packet data. 3. An i-th set of scalable object data sets composed of a plurality of N layers
The image object data, audio object data, and digital object data corresponding to the layers (i = 1..N) are multiplexed, and the i-th object data (i
= 1. . N), packetize, generate i-th packet data (i = 1... N), and further multiplex N packet data from the first packet data to the N-th packet data to obtain packet data A data multiplexing method, wherein different priorities are assigned to N packets from the first packet data to the Nth packet data. 4. An ith scalable set of object data consisting of a plurality of N hierarchies,
After packetizing image object data, audio object data, and digital object data corresponding to the hierarchy (i = 1..N), packet multiplexing is performed to generate i-th packet data (i = 1..N); The same first packet number for identifying the packet is assigned to the packet of the object indicating the same component having a different hierarchy and having a scalability relationship, and N packets from the first packet data to the N-th packet data are assigned. When two packet data are further multiplexed to obtain packet data, a second packet number for identifying the packet is added to N packets from the first packet data to the N-th packet data. Data multiplexing method. 5. A scramble device and a descramble device, wherein the scramble device receives multiplexed data multiplexed by the data multiplexing method according to claim 1 as input and object data corresponding to each scalability layer. For each, set a scramble mode, perform scramble, output, the descramble device,
When the scramble data from the scramble device is input and a scramble key is provided, the scramble mode is set for each object data corresponding to each scalability layer, and descrambling is performed. apparatus. 6. A multiplexed data multiplexed by the data multiplexing method according to claim 1 is input, a scramble mode is set for each object data corresponding to each scalability layer, and scrambling is performed. Scrambler to do. 7. A scrambling mode is set for each object data corresponding to each scalability layer when a signal from the scrambling device according to claim 6 is input and a scrambling key is provided, and descrambling is performed. Characteristic descrambling device. 8. A multiplexed data reproducing apparatus which receives data generated by the data multiplexing method according to claim 1, and reproduces an image, audio, or other digital data. 9. A recording medium for recording multiplexed data generated by the data multiplexing method according to claim 1, 2, 3, or 4. 10. A transmission medium for transmitting multiplexed data generated by the data multiplexing method according to claim 1.