JP2002094995A - Data processing method, data processing unit and data processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To surely obtain time information required for reproduction from data streams including object streams without narrowing a frequency band of a transmission channel nor being affected by the protection of intellectual properties. SOLUTION: Time management information of other object is generated on the basis of time management information provided to a specific stream having a smaller information quantity than that of at least voice data or not being an object of protection of intellectual properties in the case of transmitting one data stream including streams each having prescribed information, and a receiver side separates each stream from the data stream and interprets the time management information of other object on the basis of the time management information extracted from the specific stream.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data processing method, a data processing apparatus and a data processing system, and more particularly to a data processing method and a data processing method for handling one data stream including a plurality of object streams each having predetermined information. The present invention relates to an apparatus and a data processing system.

[0002]

2. Description of the Related Art In recent years, there have been various methods for encoding various data such as moving images and audio, treating each encoded data as an object, and combining these so-called multimedia data into a single bit stream. Is being considered. Among them, the international standardization organization IS
The so-called MPEG-4 (Moving Picture Experts)
Group phase 4) is being standardized.

On the receiving side (reproducing side) of this MPEG-4, for example, audio and an object such as a moving image scene are reproduced in association with each other. Such MPEG-
In the 4 system, the received bit stream is used because of the property that data is treated as an object.
(Data stream) can be easily reorganized individually for each object.

[0004] The above-described MPEG-4 data stream has a function of transmitting and receiving several video scenes and video objects independently on a single stream, unlike conventional multimedia streams. . Similarly, audio has a function of independently transmitting and receiving several objects on a single stream.

[0005] Here, referring to FIG. 7 and FIG.
No. 4 reproducing (receiving) apparatus and its data processing mechanism will be described. FIG. 7 is a diagram showing a data structure in an MPEG-4 data stream, and FIG.
FIG. 2 is a block diagram illustrating a schematic configuration of a PEG-4 playback device.

FIG. 7 shows an MPE in which a data stream (object stream) of each object is multiplexed.
G-4 data stream (MPEG-4 stream)
201 is shown. The MPEG-4 stream is input to a multiplexer 302 via a transmission path 301 in FIG. 8, and is separated into an encoded stream 202 of each information (object) by the demultiplexer 302. 30
Reference numerals 3, 304, 305, and 306 denote objects (in this example, including audio, moving image, scene description data, and object description data) in units of the access unit 203 (FIG. 7), which is a unit of processing for decoding time management and synchronization. Is a buffer for a decoder to which the above data stream is input, and the buffer 303 for the decoder
In 306, decoders 307 and 30 provided for each object after temporarily holding the access unit
8, 309 and 310 respectively.

The decoders 307 to 310 decode the access units, respectively, and convert the composition unit 205, which is a processing unit at the time of synthesis, to the synthesis memory 3
11, 312, 313, 314 are supplied for each object. In the combining memories 311 to 314, the composition unit 205 is temporarily stored and then output to the combiner 315. The synthesizer 315 stores each decoded object (each composition unit 205) in the memory 31.
3 and outputs multimedia playback information such as a playback scene.

Next, decoding (decoding) in the reproducing apparatus
The processing timing of composition and the mechanism of time management at that time will be briefly described with an example.

[0009] The time management information 204 in FIG. 7 is added for each access unit 203. The time management information 20
Reference numeral 4 includes an object reference clock from which a time reference of the object is set, a decoding time stamp indicating decoding timing, and a synthesis time stamp indicating synthesis timing. Here, the object time reference is a time reference set for each object. In the reproducing apparatus, the object time reference is set based on the object reference clock added to the access unit.

Here, a map is set for associating the object time reference with the system time reference which is the reference time of the reproducing apparatus. Decoder buffer 303
The access unit 203 inputted and held in 306 refers to the time of the decoding time stamp mapped on the system time reference, and each object is decoded at that time. The objects decoded in this way are used as composition units 205 as compositing memories 311 to 311.
14 and temporarily stored. In the synthesizer 315, the composition units stored in the memories 311 to 314 are synthesized at the time with reference to the time of the synthesis time stamp mapped on the system time reference, and output to the output device 316. become. Output device 3
The combined scene is displayed on the display device 16.

Here, in the system as described above, when processing a data stream having a plurality of object streams, each object has a time reference. Therefore, in such a case, synchronization between objects is not guaranteed in the process of synthesizing the composition unit. If the synchronization between the objects is not guaranteed, the display timing or the like will naturally be shifted.

[0012] Such a problem arises when the transmitting apparatus generates an MPEG-4 stream, even if a system time reference which is a time reference of the transmitting apparatus exists, the MPEG-4 stream is configured. Even if the decoding time stamp and the synthesis time stamp of each object are generated by matching or synchronizing the object time reference of each object with the system time reference, it is necessary to restore the system time reference in the reproducing apparatus. This occurs because there is no mechanism for transmitting the required object reference clock.

[0013] To cope with this problem, generally, a time reference is set for a sound object in a transmitting apparatus, and the time reference for the sound object is set as a time reference of the system. That is, decoding time stamps and synthesis time stamps of objects other than voice are generated in synchronization with the object time reference (system time reference) of the voice and added to the access unit.

In the reproducing apparatus, the audio object time reference is restored based on the audio object reference clock. The audio object time reference is used as the system time reference in the playback apparatus, and the decoding time stamp and the synthesis time stamp of each object described above are interpreted based on the system time reference to decode the entire multimedia stream, Restoration and synthesis synchronization were ensured.

[0015]

As described above, in the current general method, synchronization of decoding and reproduction (synthesis) is guaranteed based on time information of audio data.
If the audio object stream itself does not exist, or if the audio object stream cannot be received or cannot be received for some reason, decoding or synthesizing another object stream, that is, it is impossible to restore the multimedia data itself Had become.

More specifically, when a program composed of a scene such as a moving image only or a still image becomes unreproducible, an audio object stream fails to be received, or an audio object stream cannot be received for any reason, the multimedia scene It means that the whole becomes unreproducible.

To solve the above problem, it is conceivable to forcibly add an audio object stream irrelevant to a scene. In this case, it is possible to use the time information of the audio object stream to perform time management of decoding of the access unit and composition of the composition unit. If the reproduction level of the audio object used for the time management is suppressed to a low level, the reproduction itself is performed. Becomes possible.

However, in the above-mentioned method, a data stream having a relatively large data amount, ie, audio, has to be transmitted on a transmission path regardless of the original scene description. , The transmission capacity (bandwidth) of the originally required object must be reduced.

All the conventional data compression (high-efficiency coding) algorithms have been made in order to effectively use such precious transmission capacity as described below. The transmission of unnecessary useless audio objects is not preferable.

The present invention has been made in view of the above background, and performs decoding and reproduction by separating each object stream from a data stream including a plurality of object streams without narrowing the bandwidth of a transmission path. It is an object of the present invention to provide a data processing method, a data processing device, and a data processing system capable of obtaining time information required at the time from a data stream.

[0021]

In order to solve such a problem, a data processing method and a data processing apparatus according to the present invention are provided.
As one aspect, when one data stream is generated from a plurality of object streams each having predetermined information, time management information is added to at least a specific object stream having an information amount smaller than audio, It is characterized in that time management information of another object is generated based on time management information given to a specific object stream.

In one aspect of the data processing method and data processing apparatus of the present invention, each object stream is separated from a data stream including a plurality of object streams each having predetermined information, and these objects are combined. When reproducing the synthesized information,
In the data stream, at least time management information from a specific object stream having a smaller amount of information than audio is extracted, and time management information of another object is extracted based on the time management information extracted from the specific object stream. Is interpreted.

In the above manner, the time management information can always be added without narrowing the bandwidth of the transmission path, and if the specific object stream can be restored on the receiving side, each object stream can be reliably separated. As a result, time information necessary for decoding and reproduction can be obtained, and the time information can be obtained from the data stream of the object faithful to the original time.

According to another aspect of the data processing method and the data processing apparatus of the present invention, when one data stream is generated from a plurality of object streams each having predetermined information, the object is protected by intellectual property. The present invention is characterized in that time management information is added to a specific object stream for an object to be changed and time management information of another object is generated based on the time management information added to the specific object stream.

Further, as another aspect, the data processing method and the data processing apparatus of the present invention separate each object stream from a plurality of object streams each having predetermined information and a data stream including intellectual property management information, When reproducing the synthesized information obtained by synthesizing each of these objects, based on the intellectual property management information, at least a part of the plurality of object streams is controlled to be reproduced, and in the data stream, protection of intellectual property is protected. Extracting time management information from a specific object stream for an object that is not a target and interpreting time management information of another object based on the time management information extracted from the specific object stream; .

With the above arrangement, even if some of the objects become unreproducible from the viewpoint of intellectual property protection, the time management information can always be added, and the specific object stream is transmitted to the receiving side. If it can be restored with, each object stream is surely separated, and time information necessary for decoding and playback is obtained,
It can be obtained from the data stream of the object that is true to the original time.

The various objects and features of the present invention are described in detail in the following “Embodiments of the invention”.

[0028]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a data processing apparatus and a transmission apparatus constituting a data processing system according to an embodiment of the present invention, which is an apparatus for outputting an MPEG-4 data stream. In the figure, reference numerals 101, 102, and 103 denote input terminals for a moving image signal, a still image signal, an audio signal, and the like.
4. Still image encoder 105, audio encoder 1
06, and is subjected to high-efficiency (compression) encoding by an optimal encoding method for each object.

Here, the moving image is, for example, a well-known MPEG2
Method and H263 method, a still image is subjected to a well-known discrete wavelet transform, and then subjected to entropy coding such as quantization and arithmetic coding to perform high-efficiency coding. Is highly efficient coded by well-known CELP (Code Excited Linear Prediction) coding, transform domain weighted interleave vector quantization (TWINVQ) coding, or the like.

Reference numeral 107 denotes an object description data encoder, which encodes object description data such as animation such as 3D graphics by encoding suitable for each description data format and outputs it. Also, 1
Reference numeral 08 denotes an encoder for scene description data, which is a BIF obtained by modifying a well-known VRML as information (scene description information) for synthesizing a scene from the above objects.
Output S (Binary Format for Scenes). As will be described later, the reproduction side synthesizes objects based on the BIFS description and restores the scene.

The data streams output by the moving picture encoder 104, the still picture encoder 105, the audio encoder 106, the object description data encoder 107, and the scene description data encoder 108 are buffers 111, 112, 113, 114 and 115, respectively. To be combined into one data stream by the multiplexer 131. Here, the decoding time stamp and the composite time stamp represented based on the time reference of each object reference are generated by each of the encoders 104 to 108, and are stored in the buffers 111 to 1 on an access unit basis.
15 once.

The time information data generator 128 sets a system time reference of the transmitting apparatus, and converts time stamp conversion circuits 121, 122, 1 provided for each object.
23, 124, and 125. Each conversion circuit 121
As is clear from the above description, the time stamps (decoding time stamp and composite time stamp) stored in the buffers 111 to 115 are mapped on the system time reference based on the time reference of each object. Each buffer 1 is converted by referring to the time of the synthesis time stamp.
Overwrite on the time stamp of the access unit stored in 11-115.

FIG. 2 is a diagram for explaining the data conversion performed by the transmitting side apparatus in FIG. 1. As shown, a decoding time stamp (DTS) and a combining time stamp (CTS) set on the time reference of each object are shown. Is associated with the system time reference from the time information data generator 128.

On the other hand, the time information data generation unit 128
An object stream providing only time information as described later is formed and supplied to the buffer 130. The multiplexer 131 appropriately superimposes the object streams obtained through the buffers 111 to 115 on the object stream dedicated to time information obtained through the buffer 130 in a time-division manner, and outputs an MPEG-4 stream.

The MPEG4 stream is supplied to an output interface (output IF), and after being converted into an optimal data format by a transmission line 133, the transmission line 1
33. Here, the transmission path 133 is a data path such as various networks and a computer bus. In the present embodiment, the transmission path 133 is a network to which an MPEG-4 stream is input. Of course, the transmission path here is
CD-ROM, DVD-ROM, D
It also includes a recording medium such as a VD-RAM and an interface with the recording medium.

Next, the data structure of an object stream that provides only time information used in this embodiment will be described with reference to FIG. The data structure itself is defined in ISO / IEC 14966-1 MPEG-4 Systems.

The elementary stream 601 is transmitted in the access unit layer described above with reference to FIG.
The access unit 602 is divided into SL-PD
It is packetized as U (Sync Layer Protocol Data Unit). The SL-PDU includes an SL-PDU header 603 and an SL-PDU payload 604.
Various information including time management information exists in the L-PDU header 603.

Each access unit 602 can be divided into a plurality of packets, but the head of each access unit 602 is always placed at the head of the SL-PDU payload 604. SL-PDU header 603, SL-PD
For the U payload 604, the packet length is specified to be an integral multiple of 8 bits within the limit. SL-PDU header 60
If the packet length of 3 is not a multiple of 8 bits, "0" is padded. SL-PDU payload 60
Even when the packet length of 4 does not become an integral multiple of 8 bits, padding with "0" is performed, and the information is described in the SL-PDU header 603.

Here, the packet length of the SL-PDU payload 604 is not specified. Therefore, the data amount of the payload is set to the minimum in consideration of the object stream that provides only the time information as long as only the time information can be provided to the playback device. By doing so, it is possible to configure an effective system that does not impose a load on the transmission path by the amount of the audio data as compared with the case where the time information of the unused audio object stream is used. it can.

Although only a data stream dedicated to time information is described in this embodiment, the same effect can be obtained by streaming this time information together with other data having a smaller data amount than audio data. Needless to say, this is obtained. Therefore, although a greater effect can be obtained by providing a data stream dedicated to time information, the present invention uses time information together with information having a smaller data amount than audio data having a relatively small data amount in various objects. It should be understood that they are streamed and sent in another object.

The main information included in the SL-PDU header 603 in FIG. 3 will be described below.

First, the access unit start Fl in FIG.
ag 605 is data indicating that the head data of the access unit is at the head of the payload. access Unit en
d Flag 606 is data indicating that the last data of the access unit is present in the payload. The OCR Flag 607 is data indicating the presence or absence of an object reference clock.
The padding Flag 608 is data indicating the presence or absence of padding of the payload. padding Bits 609 is data indicating the number of bits of the padding. Packet Sequence N
umber 610 is a counter value that increases for each SL-PDU used for discarding and controlling the order of the SL-PDU. object
Clock Reference 611 is data of an object reference clock. The decoding time stamp flag 612 is data indicating the presence or absence of a decoding time stamp. composition T
The ime Stamp 613 is data indicating the presence or absence of a synthesis time stamp. The decoding time stamp 614 is data of a decoding time stamp. The composition time stamp 615 is data of a composition time stamp. The access unit length is data indicating the data length of the access unit in bytes. degradation Priority 617 is the SL-PD
This is data indicating the priority of U.

FIG. 4 shows a data processing apparatus and a data processing system according to an embodiment of the present invention, and a reproducing apparatus (decoding apparatus) for receiving and reproducing the MPEG-4 data stream transmitted by the above-mentioned transmitting apparatus. Is a block diagram showing a configuration of the data transmission device. A data stream input from a transmission line 135 is converted by an input interface (input IF)
Converted to a data format that can be processed, MPEG-4
The data is input to the demultiplexer 138 as a stream.

In the demultiplexer 138, M
The PEG-4 stream is separated into encoded streams of each object (moving image, still image, audio, object description data, scene description data, object dedicated to time information), and the encoded stream is divided into access units for each stream. , Decoder buffers 141, 142, 143, 144, 145 provided for each object
Is input to

Here, the time information dedicated object stream is similarly stored in the time information buffer 1 for each access unit.
The object reference clock supplied to the access unit and added to the access unit is extracted. The access unit of the stream dedicated to time information is supplied to the system time reference control unit 150 together with the object reference clock. The control unit 150 sets a time reference for the time information-dedicated object based on the input object reference clock. Next, the object time reference is made to match the system time reference which is the time reference of the playback device. In the subsequent processing, the system time reference becomes the only time reference, and all processing is performed by the system time reference control unit 150 according to this time reference.

The control unit 150 controls the decoder buffer 14
The decoding time stamp and the combined time stamp of each object added to each access unit stored in 1, 142, 143, 144, and 145 are extracted. Then, at the time according to the extracted decoding time stamp, the decoder buffers 141, 142, 143, 144, 145
Each access unit stored in the decoder unit 1
51, 152, 153, 154, and 155, and each decoder section 151, 152, 153, 154, 1
From 55, a composition unit for each object is output.

Each composition unit is stored in the synthesizing buffers 161, 162, 163, 164, 165, respectively.
Reference numeral 50 denotes buffers 141 to 145 and a decoder 151 based on the decoding time stamp based on the system time reference.
155, and further, by controlling the buffers 161 to 165.

The control unit 150 similarly sets the synthesizing buffer 16 at a time according to the extracted synthesizing time stamp.
The buffers 161 to 165 and the combining unit 171 are controlled so that the combining units 171, 163, 164, and 165 combine the respective composition units.

In the output interface (IF) 172, the synthesizing unit 1 according to the decoded scene description data.
The object synthesized at 71 is converted into a data format according to an output device 173 such as a display, a speaker, and a printer. As described above, after extracting the object reference clock from the object stream dedicated to time information, an object time reference is set from the object reference clock, and the object time reference is matched with the system time reference which is the reference time of the playback device. By interpreting the decoding and synthesizing timings of other objects on the basis of this system time reference, even if the multimedia information without sound is composed of scenes such as moving images or still images only, it is possible to obtain the time The output can be performed by the output device without any deviation.

In particular, when reproducing such multimedia information without sound without time lag, the data amount of the MPEG-4 stream to be transmitted can be made smaller than before.

Next, in the reproducing apparatus shown in FIG. 4, most parts including the system time reference control unit 150 can be constituted by software. Hereinafter, processing by software in the playback device will be described with reference to the flowchart in FIG.

As shown in step 701 of FIG. 5, the playback device determines whether the M containing the time information-dedicated object stream
When the PEG-4 stream is received, it is decomposed into each object stream and stored in each decoder buffer for each access unit.

Next, at step 702, it is checked whether or not data exists in these decoder buffers. If data exists, the process proceeds to step 703. If data exists, the process ends. Step 70
In step 3, the object reference clock (OCR) is obtained from the access unit (AU) of the time information-dedicated object stream, and in step 704, the object time reference (OCR) is obtained from the object reference clock.
TB) is set.

In step 705, the object time reference is made to match the system time reference (STB) which is the time reference of the playback device. In step 706, the access unit is decoded according to the decoding time stamp (DTS) of the access unit of each object stored in the decoder buffer, a composition unit (CU) is generated, and this is stored in the synthesis buffer. I do.

In step 707, the composition units of the objects stored in the combining buffer are combined and output according to the combining time stamp of the access unit. When the processing in step 707 ends, the flow returns to step 702, and it is determined again whether or not data exists in the decoder buffer. By executing the routine as described above, it is possible to reproduce each object without time lag in the reproducing apparatus as described above.

FIG. 6 is a block diagram showing another example of a reproducing apparatus which constitutes a data processing apparatus and a data processing system according to another embodiment of the present invention and receives and reproduces an MPEG-4 stream from the transmitting apparatus shown in FIG. It is a diagram, as described in Japanese Patent Application No. 10-295937, filed by the present applicant, in addition to the above-described object streams (including an object stream dedicated to time information) as an MPEG-4 stream. Intellectual Property Management (IPMP: Inte
(Llectual Property Management and Protection) information.

In FIG. 6, blocks having the same operations as those in FIG. 4 are denoted by the same reference numerals, and a detailed description thereof will be omitted.
Each object from the four streams is supplied to each of the decoder buffers 141 to 145 and the time information buffer 140 on an access unit basis, and the IPMP information of each object included in the IPMP stream is transmitted to the IPM stream.
This is supplied to the P control unit 180.

The IPMP control unit 180 extracts the extracted IPMP
Based on the information, it is determined whether or not each object is an object that can be reproduced in terms of intellectual property. In the case of an object protected by intellectual property, the playback device checks (authenticates) whether or not the playback device is licensed for the intellectual property using a back channel, and checks whether there is any problem in intellectual property. Alternatively, only when the intellectual property is licensed, the output of data (access unit) from the decoder buffers 141 to 145 to the decoder units 151 to 155 is permitted. If the intellectual property is licensed, transfer of data from the decoder buffers 141 to 145 to the decoder units 151 to 155 is prohibited.

As described above, the decoder buffer 1 for storing the access units and extracting the time management information such as the decoding time stamp and the synthesis time stamp of each access unit.
By performing the control of 41 to 145 by the IPMP control unit 180 and the system time reference control unit 150, especially
IPMP control can be performed without providing a separate control target for PMP.

In the above embodiment, even if the access unit of the audio object stream is discarded by the IPMP control unit 150 in the audio buffer for decoder 143, the access unit of the object stream dedicated to the time information can be used to obtain the object reference clock,
Since the object time reference is obtained and the system time reference can be obtained from the object time reference, the time management in the playback device does not break down due to regulation by intellectual property. That is, in the present embodiment, an object other than an object that can be protected by intellectual property, such as a sound or an image, is provided with time information, and the transmitting side generates a time stamp of each object using the time information as a system time reference. On the reproduction (reception) side, the reproduction time of each object can be reliably managed and reproduced by generating a system time reference from a time information object that is not protected by the intellectual property.

[0061]

As described above, according to the data processing method, the data processing apparatus, and the data processing system of the present invention, the data processing method, the data processing apparatus, and the data processing system do not reduce the bandwidth of the transmission path or are affected by the protection of intellectual property. Without separating each object stream from a data stream containing a stream of multiple objects without fail.
Time information required for performing reproduction can be obtained from the data stream.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a data processing device and a transmission device constituting a data processing system according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining data conversion performed by a transmission-side device in FIG. 1;

FIG. 3 is a diagram showing a data configuration of an object stream that provides only time information handled by the apparatus of FIG. 1;

FIG. 4 is a block diagram showing a configuration of a data processing apparatus and a data processing system according to an embodiment of the present invention, and a configuration of a reproducing apparatus which receives and reproduces a data stream transmitted by the transmitting apparatus of FIG. 1; .

FIG. 5 is a flowchart illustrating processing by software when most of the device in FIG. 4 is realized by software;

FIG. 6 shows a data processing apparatus and a data processing system according to another embodiment of the present invention.
FIG. 4 is a diagram showing another example of a playback device that receives and plays back a-4 stream.

FIG. 7 is a diagram showing the structure of data in an MPEG-4 data stream.

FIG. 8 is a block diagram illustrating a schematic configuration of a general MPEG-4 playback device.

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C059 MA00 MA24 MB27 ME11 PP01 PP04 RB06 RB09 RC04 RC11 RC32 RC33 SS13 SS30 UA02 UA05 5C063 AB03 AB07 AC10 CA16 5K034 HH05 HH12 HH15 HH18 HH65 MM01 MM15 MM39 PP05 Q

Claims (21)

[Claims] 1. A data processing method for generating one data stream from a plurality of object streams each having predetermined information, wherein time management information is added to a specific object stream having at least a smaller amount of information than audio. A data management method for generating time management information of another object based on the time management information attached to the specific object stream. 2. A data processing method for generating one data stream from a plurality of object streams each having predetermined information, the method comprising: providing a specific object stream for an object not subject to intellectual property protection; A data processing method comprising: adding management information; and generating time management information of another object based on the time management information assigned to the specific object stream. 3. The data processing method according to claim 2, wherein the data stream includes intellectual property management information. 4. The data processing method according to claim 1, wherein the specific object stream is an object stream dedicated to time management information. 5. The plurality of object streams,
5. The data processing method according to claim 1, wherein the data processing method includes at least a video object stream and an audio object. 6. A means for forming a plurality of object streams each having predetermined information, and a means for forming a specific object stream having at least a small amount of information compared to audio and to which time management information is added. Means for generating time management information of another object based on the time management information assigned to the specific object stream, and assigning the time management information to each object stream; and Means for generating one data stream including a specific object stream. 7. A means for forming a plurality of object streams each having predetermined information, and a means for forming a specific object stream provided with time management information for objects which are not subject to protection of intellectual property Means for generating time management information of another object based on the time management information given to the specific object stream, and giving the time management information to each object stream; and Means for generating one data stream including the specific object stream. 8. The data processing apparatus according to claim 7, wherein the specific object stream is an object stream dedicated to time management information. 9. The means for forming the plurality of objects includes means for encoding predetermined information, and the time management information given to the other object stream includes information indicating a time at which the information is decoded. 9. The data processing device according to claim 6, wherein 10. The data processing apparatus according to claim 9, wherein the time management information added to the other object stream includes information indicating a time at which the information is combined with the other information. 11. A data processing method for separating each object stream from a data stream including a plurality of object streams each having predetermined information and reproducing synthesized information obtained by synthesizing each of the objects, wherein: Extracting time management information from a specific object stream having at least a smaller amount of information than audio, and interpreting time management information of other objects based on the time management information extracted from the specific object stream. Characteristic data processing method. 12. A data processing method for separating each object stream from a plurality of object streams each having predetermined information and a data stream containing intellectual property management information, and reproducing combined information obtained by combining these objects. Controlling a reproduction of at least a part of the plurality of object streams based on the intellectual property management information, and a specific object stream targeted at an object that is not a target of intellectual property protection in the data stream. A time management information extracted from the specific object stream and interpreting time management information of another object based on the time management information extracted from the specific object stream. 13. The data processing method according to claim 11, wherein the specific object stream is an object stream dedicated to time management information. 14. The data processing method according to claim 11, wherein the plurality of object streams include at least a moving image object stream and an audio object. 15. A means for receiving a data stream including a plurality of object streams each having predetermined information; a means for separating each object stream from the received data stream; and restoring each object from each object stream Means for reproducing synthesized information obtained by synthesizing the restored objects; and means for extracting time management information from a specific object stream having at least a smaller amount of information than audio in the data stream. And a means for controlling a restoration timing of each object and a synthesis timing of each object based on time management information extracted from the specific object stream. 16. The data processing apparatus according to claim 15, wherein the specific object stream is a stream relating to an object that is not subject to intellectual property protection. 17. The data stream includes intellectual property management information, and further comprising means for controlling reproduction of an object from at least a part of the plurality of object streams based on the intellectual property management information. 17. The data processing device according to claim 15, wherein: 18. A buffer for temporarily storing a part of each of the object streams, and control of reproduction of the object based on the intellectual property management information controls writing or reading of data to or from the buffer. The data processing device according to claim 17. 19. The apparatus according to claim 18, wherein each of the object restoration timings is controlled using the buffer.
A data processing device according to claim 1. 20. A system for transmitting and receiving one data stream including a plurality of object streams each having predetermined information, the system comprising: a time management unit for a specific object stream having at least a small amount of information compared to audio; Information, time management information of another object is generated based on the time management information given to the specific object stream, and the data stream is transmitted to the transmission path. Data processing, separating a stream, extracting time management information from the specific object stream, and interpreting time management information of another object based on the time management information extracted from the specific object stream. system. 21. A system for transmitting and receiving one data stream including a plurality of object streams each having predetermined information, wherein time management information is added to a specific object stream not subject to intellectual property protection. Then, time management information of another object is generated based on the time management information given to the specific object stream, and the data stream is transmitted to the transmission path. On the receiving side, each object stream is separated from the data stream. And controlling the reproduction of at least a part of the plurality of object streams based on the intellectual property management information extracted from one of the plurality of object streams, and extracting the time management information extracted from the specific object stream. Time management of other objects based on A data processing system characterized by interpreting information.