JPH10320914A - Code-recording apparatus, method for multiplexing code - Google Patents

Abstract

PROBLEM TO BE SOLVED: To constitute a multiplex bit stream of a sound compression code changing a bit rate and an animation image compression code, so as to reduce a capacity of a buffer of a reproducing apparatus. SOLUTION: Concurrent video and audio code strings are divided to packets, multiplexed and recorded to a recording medium 90 in the code-recording apparatus. The apparatus has a multiplex means 410 which compares time stamps indicating a reproduction time and/or a decode time of video frames and audio frames to make the video frames and audio frames concurrent.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a code for a plurality of pieces of information (= a plurality of pieces of information having the same time property) that are required to have the same time in reproduction output, such as a moving image and a sound. And a code multiplexing method for multiplexing and recording on a medium. The present invention also relates to a code recording apparatus and a code multiplexing method for starting multiplexing any one of a plurality of pieces of information requiring the same time from an arbitrary position of other information and recording the multiplexed information on a medium.

[0002]

2. Description of the Related Art As a method for compressing and encoding digital moving images and audio, for example, MPEG standardized by ISO
And H.264 recommended by CCITT. 261. In these methods, moving pictures are encoded at a high compression rate while maintaining high image quality by interweaving intra-image compression and inter-image compression. Here, the compression within an image unit is a compression mode that can be decoded using only the compressed data, and the compression between image units is a compression mode that performs encoding using a correlation with other image data.

In intra-picture compression, all macroblocks in the picture are subjected to two-dimensional discrete cosine transform (DCT) as they are to obtain a spatial frequency component, which is subjected to matrix division called quantization. In this method, a frequency component is divided by a matrix having a predetermined value. The frequency component can be obtained by multiplying the matrix by the decoder and the quantized data. In addition, it becomes possible to reduce the precision below the matrix value. In addition, utilizing the fact that human visual characteristics are insensitive on the high frequency side,
More codes are assigned to the low frequency side than to the high frequency side.

In the inter-image compression, a motion vector indicating a displacement from a reference image is obtained for each macroblock.
The area closest to the macroblock is detected. Further, two-dimensional discrete cosine transform (DCT) and quantization are performed on the difference between the region and the macroblock data, and the difference data and the motion vector are encoded.

Therefore, the amount of bits generated in a moving picture coding system such as MPEG greatly varies depending on the complexity of the picture, the size of the motion, and the like. For this reason, a method of adjusting the amount of generated bits by adjusting a multiplier for a quantization matrix called a quantization width at the time of quantization is performed.
However, this method has a problem that the image quality is degraded in a complex image or a scene with a large motion, and conversely, the image quality is unnecessarily high in a simple image or a scene with a small motion.

For this reason, the moving picture coding method uses GO
A variable bit rate method that suppresses a change in image quality by allowing a change in the amount of generated bits in P units is being adopted. In this method, a buffer memory is provided in a stage preceding the decoder on the reproducing apparatus side, thereby absorbing the variation in the amount of generated bits. For example, Japanese Patent Laying-Open No. 4-326687 discloses a device that encodes a moving image at a variable bit rate in accordance with the MPEG standard and limits the maximum bit rate. The fact that a variable bit rate is more desirable in order to obtain a high compression rate while ensuring the quality of reproduced output also holds true when compressing and encoding audio information.

[0007]

When a moving image is accompanied by sound, compressed data of the moving image (video) and compressed data of sound (audio) are multiplexed. For example, MPE
In the G system, as shown in FIG. 2, each of the video compression code data and the audio compression code data is divided into a number of packets of an appropriate length, and a packet header indicates whether the data is video or audio. Information is added and multiplexed.

[0008] Each of the moving image and the audio has a reproduction unit (one video frame for the moving image and one audio frame for the audio) called an access unit. The packet header of the packet including the head of the reproduction unit includes: As shown in the figure, a time stamp DT indicating a decoding time
S and a time stamp PTS indicating a reproduction time are added. Note that no time stamp is added to the packet header of a packet that does not include the head of the access unit. If the same packet includes the beginning of two or more access units, only the time stamp corresponding to the preceding access unit is added to the packet header of the packet. Here, time stamp D
TS and PTS are both MPs taken as examples.
This is time information according to the EG standard.

[0009] The DTS is an order of the video frames in the bit stream and the order of the video frames in the reproduction output, because the MPEG standard allows forward predicted frames, backward predicted frames, and bidirectional predicted frames. Has been added in consideration of the fact that That is, in the MPEG bit stream, the I picture and the P picture are added in consideration of being arranged so as to precede the B picture that refers to the I picture and the P picture.

On the decoder side, each packet is stored in a buffer memory, and the DTS is used as a system synchronization signal STC (Sy).
(Stem Time Clock), the data in the buffer memory is read and decoded, and the PTS is set to ST.
By performing playback output when the value matches C, synchronized playback of video and audio is realized.

Here, a description will be given of synchronous reproduction by a digital video disk decoder in which video and audio digital data are recorded. FIG. 11 shows an example of a decoder for a digital video disc.
Data read at a predetermined rate from the recording medium 90 is:
The data is decoded by the DVD decoder 101 and stored in the disc rate control RAM 103. When the disk rate control RAM 103 becomes full, the recording medium 90
Reading from is suspended. Disc rate control R upon request from MPEG system decoder 105
When data is transferred from the AM 103 to the MPEG system decoder 105 and can be stored in the disc rate control RAM 103, reading from the recording medium 90 is restarted.

As described above, the disc rate control RAM 103 functions as a buffer for adjusting the difference between the reading speed from the recording medium 90 and the decoding speed of the video decoder 111 and the audio decoder 113 at the subsequent stage. The difference fluctuates with the fluctuation of the decoding speed in the video decoder 111 and the audio decoder 113. This fluctuation amount becomes particularly large when the compressed video data has a variable bit rate as shown in FIG. 15, that is, when the data amount changes in GOP units. FIG. 14 shows a GOP
A case where the data amount in units is a fixed bit rate is shown. Even in this case, the data amount in a micro level such as a frame unit varies.

The above-mentioned disc rate control RAM
103 can be shared with the work RAM 101a of the DVD decoder 101 and / or the AV rate control RAM 105a described below.

MPEG transferred from the disc rate control RAM 103 to the MPEG system decoder 105
The system bit stream is stored in the AV rate control RAM 105a. This data is analyzed by the MPEG system decoder 105, separated into compressed video data and compressed audio data, and stored in the AV rate control RAM 105a.

The compressed video data and compressed audio data stored in the AV rate control RAM 105a are read out at the timing indicated by each DTS, sent to the video decoder 111 or the audio decoder 113, and output to the video decoder 111 or the audio decoder 113. Decoded to data. The decoded video data is stored in the RAM 111a, and the decoded audio data is stored in the RAM 111a.
Stored in 3a. The video data stored in the RAM 111a and the audio data stored in the RAM 113a are read and output at the timing indicated by each PTS. Thus, synchronous reproduction of video data and audio data is realized.

It is assumed that the DTS or PTS giving the above read timing is extracted and stored by the analysis of the MPEG system decoder 105. This storage is performed, for example, in the AV rate control RAM 105a.
Or a dedicated register or a register of the controller 131.

Since the processing in the decoder is performed as described above, the time positions of the DTS are close (preferably at the same time).
When the compressed video data and the compressed audio data are A
When the data is stored almost simultaneously in the V rate control RAM 105a, the reading from the AV rate control RAM 105a is also substantially simultaneous,
The time during which unprocessed compressed data stays in the memory 105a is also shortened, and the AV rate control RAM 105a requires a relatively small-capacity memory.

It is an object of the present invention to allow compressed video data and compressed audio data data to be stored almost simultaneously in an AV rate control RAM of a decoder, in other words, to multiplex them so as to have the same time. It is.

FIG. 11 shows a case of a decoder for a digital video disk on which video and audio digital data are recorded.
A decoder (a decoder for reading and reproducing a video-CD in which data of the MPEG standard is recorded in a video-CD format) or a DVD decoder (MP
Even in the case of a decoder that reads and reproduces a DVD in which EG2 standard data is recorded in the DVD format, decoding is performed in the same manner as in the case of FIG.

An object of the present invention is to reduce the capacity of a memory having the function of an AV rate control RAM (memory for storing compressed video data and compressed audio data) in the decoder. Also, for this purpose,
It is intended to be achievable with a simple configuration. It is another object of the present invention to be able to construct a bit stream that meets the above-mentioned purpose in real time. It should be noted that the present invention can be implemented in real time or not in real time.

In view of the fact that various high-density recording media capable of recording / reproducing moving images and audio have been proposed, the present invention compresses and encodes moving images and audio in real time as described above. An object of the present invention is to enable each code to be multiplexed and recorded on a recording medium so as to be substantially equal to a ratio of an instantaneous bit rate. That is, an object of the present invention is to enable multiplexed recording so as to have the above-mentioned time synchronization.

Another object of the present invention is to enable multiplexing and recording of audio and subtitles from an arbitrary position in a moving image. It is another object of the present invention to enable subtitles and other audio to be multiplexed and recorded from an arbitrary position of multiplexed video and audio data. It is another object of the present invention to multiplex and record compression-encoded data of a moving image and audio and to multiplex and record a desired subtitle or another audio from an arbitrary position. In other words, an object of the present invention is to enable multiplexing and recording of different information from an arbitrary position of a plurality of types of information requiring the same time.

[0023]

A first aspect of the present invention is a code recording apparatus for multiplexing a plurality of types of information code strings having the same time and recording the same on a recording medium. Calculating means for obtaining a temporally-preceding relationship that is not restricted to a code sequence with respect to an access unit of each code sequence based on position information indicating the position of each of the code sequences; And a multiplexing means for multiplexing in time sequence.

The position information includes, for example, an absolute time (hour, minute, second), a code indicating the order of frames, and the like.
“Determining based on position information” may be determined directly from the absolute time, or may be determined from the assigned time of one frame and the frame order that differ for each type of information. An access unit is a unit of data obtained by encoding a frame, and is a unit of encoding and decoding. "Not tied to a code string" means to obtain within a code string and between code strings. "Packetization" may be not only packetization but also packing. “Muxing based on the relationship” refers to multiplexing with reference to the data obtained by the arithmetic means, and multiplexing the data obtained by the arithmetic means with reference to the data further processed and added at the time of packetization. In the case where such data is a packet that is not added at the time of packetization, it is estimated from additional information on the packet including the access unit positioned immediately before and the packet including the access unit positioned immediately after in the original code string. Refers to the time position to be linked.

According to a second aspect of the present invention, in the first aspect, when the packet includes a head of an access unit, the multiplexing means sets the time management information based on the position information of the first access unit in the packet. Is generated and added as header information of the packet. The time management information is, for example, an MPEG standard time stamp.

According to a third aspect of the present invention, there is provided a code recording apparatus for multiplexing a plurality of types of information having the same time characteristic and recording the same on a recording medium, wherein the position information indicating a position in each of the code sequences is provided. And calculating means for obtaining a time-preceding relationship that is not restricted to a code string with respect to an access unit of each code string, and packetizing each of the code strings. Packetizing means for generating time management information based on the position information of the first access unit in the target packet and adding the time management information as header information of the target packet, and concatenating each packet in chronological order based on the preceding and following relationship And a multiplexing means.

"Link based on precedent and succeeding relationship" means that the connection is performed by referring to the data obtained by the arithmetic means, the connection is performed by referring to the time management information of the packet header, or the packet has no time management information. Refers to concatenation by referring to the time position estimated from the time management information of the packet including the access unit positioned immediately before and the packet including the access unit positioned immediately after in the original code string. .

According to a fourth aspect of the present invention, there is provided a code recording apparatus for multiplexing a plurality of types of information code streams having the same time characteristic and recording the multiplexed code streams on a recording medium. Packetization means for generating time management information based on the position information of the first access unit in the target packet and adding the time management information as header information of the target packet when the target packet includes the head of the access unit during the conversion, Calculating means for determining a time-preceding relationship that is not limited to the type based on the time management information with respect to the packet generated by the packetizing means;
And a multiplexing means for connecting each packet in time order with reference to the preceding and following relationship.

"Determine based on time management information"
For a packet without time management information obtained by referring to the time management information, the time of the packet including the access unit positioned immediately before and the packet including the access unit positioned immediately after in the original code string Referencing the time position estimated from the management information. “Independent of the type” means that the determination is made within the same type and between different types.

According to a fifth aspect of the present invention, there is provided a code recording apparatus for multiplexing a plurality of types of information code streams having the same time and recording the same on a recording medium. Packetization means for generating time management information based on the position information of the first access unit in the target packet and adding the time management information as header information of the target packet when the target packet includes the head of the access unit during the conversion, Extracting means for comparing the time management information of the packets generated by the packetizing means between the different types of information and extracting the packets having the time management information at the same time as a simultaneous packet set; Multiplexing means for linking packets to be transmitted and chronologically linking each set of simultaneous packets based on the time management information. A code recording apparatus for.

"Compare time management information" refers to comparing time management information. In the case of a simultaneous packet set having no time management information, a packet constituting the simultaneous packet set is immediately before in the original code string. This means comparing the time position estimated from the time management information of the packet containing the located access unit with the packet containing the immediately succeeding access unit, and the like. “Connecting the packets forming the simultaneous packet set” means connecting the packets forming the simultaneous packet set so as to be continuous. The phrase “link based on time management information” refers to linking with reference to time management information, or linking based on the estimated time position in the case of a simultaneous packet set without time management information.

According to a sixth aspect of the present invention, there is provided a code recording apparatus for multiplexing a plurality of types of information having the same time characteristic and recording the multiplexed information on a recording medium. Packetization means for generating time management information based on the position information of the first access unit in the target packet and adding the time management information as header information of the target packet when the target packet includes the head of the access unit during the conversion, Extracting means for comparing the time management information of the packets generated by the packetizing means between the different types of information and extracting packets having the time management information at the same time as a simultaneous packet set; Calculating means for obtaining a temporal pre / post relationship based on the time management information with respect to the simultaneous packet set, And multiplexing means for connecting a reference to the simultaneous packet sets the destination after relationship in time order with connecting packets formed a code recording apparatus having a.

"Determining based on time management information" means
In the case of a simultaneous packet set without time management information, which is obtained by referring to the time management information, the packets constituting the simultaneous packet set include the access unit located immediately before in the original code string and the packet including the access unit immediately thereafter. For example, it is determined based on the time position estimated from the time management information of the packet including the located access unit.

According to a seventh aspect of the present invention, there is provided a code recording apparatus which multiplexes a plurality of types of information having the same time and records them on a recording medium. When an offset position with respect to the information is specified, conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position; and Calculating means for determining a temporally-preceding relationship that is not restricted to a code string for an access unit of each code string based on offset-considered position information in the code string and position information indicating a position of the other information in the code string; and A multiplexing unit that packetizes each code string and multiplexes each packet in time order based on the preceding and following relationship.

According to an eighth aspect of the present invention, in the seventh aspect, when the packet includes a head of an access unit, the multiplexing means stores the position information or the offset-considered position information of the head access unit in the packet. This is a code recording device that generates time management information based on the information and adds it as header information of the packet.

According to a ninth aspect of the present invention, there is provided a code recording apparatus for multiplexing a plurality of types of information having the same time and recording the same on a recording medium. When an offset position with respect to the information is specified, conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position; and Calculating means for determining a temporally-preceding relationship that is not restricted to a code string for an access unit of each code string based on offset-considered position information in the code string and position information indicating a position of the other information in the code string; and Each code string is packetized, and when the target packet includes the head of the access unit at the time of packetization, the first access unit in the target packet is used. Packetizing means for generating time management information based on the position information or offset-considered position information and adding it as header information of the target packet; and multiplexing means for connecting each packet in time order based on the preceding and following relationship. It is a code recording device.

According to a tenth aspect of the present invention, there is provided a code recording apparatus for multiplexing a plurality of types of information having the same time and recording the same on a recording medium. When an offset position with respect to the information is specified, conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position; and Each packet is packetized, and when the target packet includes the head of the access unit at the time of packetization, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet, and Packetizing means for adding the packet as header information; and the time management information for the packet generated by the packetizing means. A code recording apparatus having a calculating means for calculating a non temporal antecedent relationship caught in the type, and multiplexing means for connecting a reference to each packet the destination after relation in chronological order, the based on.

An invention according to claim 11 is a code recording apparatus for multiplexing a plurality of types of code strings having the same time and recording the same on a recording medium. When an offset position with respect to the information is specified, conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position; and Each packet is packetized, and when the target packet includes the head of the access unit at the time of packetization, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet, and Packetizing means for adding the packet as header information; and time management information of the packet generated by the packetizing means. Extracting means for extracting packets having the same time management information at the same time by comparing the information as a simultaneous packet set, and connecting the packets constituting the simultaneous packet set to each other based on the time management information; Multiplexing means for connecting the packet sets in time order;
Is a code recording device.

According to a twelfth aspect of the present invention, there is provided a code recording apparatus for multiplexing a plurality of types of information having the same time and recording the same on a recording medium. When an offset position with respect to the information is specified, conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position; and Each packet is packetized, and when the target packet includes the head of the access unit at the time of packetization, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet, and Packetizing means for adding the packet as header information; and time management information of the packet generated by the packetizing means. Extracting means for comparing packets having the same time management information at the same time as a set of simultaneous packets, and a time-preceding relationship based on the time management information with respect to the set of simultaneous packets extracted by the extraction means. And a multiplexing means for connecting the packets constituting the simultaneous packet set and for connecting the simultaneous packet sets in chronological order with reference to the preceding and succeeding relations.

According to a thirteenth aspect of the present invention, in any one of the fifth, sixth, eleventh, and twelfth aspects, the extracting means cuts off a predetermined number of lower-order bits of the time management information. A code recording apparatus for performing the extraction by referring to data of a predetermined number of upper bits. The invention of claim 14 is
In any one of claims 5, 6, 11, and 12, the extracting means rounds a predetermined number of lower bits of the time management information and refers to data of a predetermined upper bit. A code recording device for performing the extraction.

For example, as shown in FIG. 3, a time unit is different between one video frame as a reproduction unit of moving image information and one audio frame as a reproduction unit of audio information.
For this reason, the start times of one video frame of a moving image and one audio frame of a sound to be associated with each other may be slightly different from each other. In view of this, in the invention of claim 13 or the invention of claim 14, the lower bits of the time management information are truncated or rounded to extract the upper bits, so that the moving picture to be associated with each other is extracted. A set of one video frame and one audio frame of audio is searched for and extracted. That is, a set of packets regarded as the same time is searched for and extracted. Note that the “time management information” includes the above-described time position estimated from the position information of the previous and next access units in the original code string when there is no time management information in the packet.

According to a fifteenth aspect of the present invention, in any one of the second, third, eighth and ninth aspects, the time management information includes decoding time information indicating a decoding time of an access unit and reproduction time information. This is a code recording device that is reproduction time information indicating a time. According to a sixteenth aspect of the present invention, in any one of the fourth, sixth, tenth, and twelfth aspects, the time management information indicates decoding time information indicating a decoding time of an access unit and reproduction time. And the calculating means obtains a temporal predecessor-relationship based on either the decoding time information or the reproduction time information. According to a seventeenth aspect, in the fifth or eleventh aspect, the time management information is decoding time information indicating a decoding time of an access unit and reproduction time information indicating a reproduction time. Is a code recording device that links each set of simultaneous packets based on either the decoding time information or the reproduction time information.

According to an eighteenth aspect of the present invention, in the code recording apparatus according to any one of the first to sixth, thirteenth, and fourteenth aspects, the plurality of types of information are moving image information and audio information. It is. The invention of claim 19 is the invention of claims 7 to 1
2. The code recording device according to any one of 2), wherein the plurality of types of information are moving image information and audio information. A twentieth aspect of the present invention is the code recording apparatus according to any one of the fifteenth to seventeenth aspects, wherein the plurality of types of information are moving image information and audio information.

According to a twenty-first aspect, in the eighteenth aspect, the code string of the plurality of types of information is a code string of compressed code data obtained by compressing moving picture information and audio information in accordance with the MPEG1 or MPEG2 standard. The access unit is a code recorder, one video frame and one audio frame. According to a twenty-second aspect, in the nineteenth aspect, the code sequence of the plurality of types of information is a code sequence of compressed code data obtained by compressing moving image information and audio information in accordance with the MPEG1 or MPEG2 standard, and the access unit includes: This is a code recording device that is one video frame and one audio frame. According to a twenty-third aspect of the present invention, in the twentieth aspect, the code sequence of the plurality of types of information is a code sequence of compressed code data obtained by compressing moving image information and audio information according to the MPEG1 or MPEG2 standard. One video frame and one audio frame,
It is a code recording device.

According to a twenty-fourth aspect, in the twenty-third aspect, the decoding time information is a DTS (Decoding Time Stamp).
And the playback time information is PTS (Presentation Tim
e Stamp), which is a code recording device.

According to a twenty-fifth aspect of the present invention, in any one of the first to sixth, thirteenth, and fourteenth aspects, the plurality of types of information are moving image information, audio information, and subtitle information. It is a code recording device. An invention according to claim 26 is the code recording apparatus according to any one of claims 7 to 12, wherein the plurality of types of information are moving image information, audio information, and caption information. The invention of claim 27 is the invention of claims 15 to 1
7. The code recording apparatus according to any one of the items 7, wherein the plurality of types of information are moving image information, audio information, and caption information.

According to a twenty-eighth aspect of the present invention, in any one of the twenty-first to twenty-seventh aspects, the data amount of each access unit in the code string of the moving image information is allowed to be different for each access unit. A code recording device. For example, according to the MPEG standard, when moving images are encoded, variable-length encoding is performed on data that has been quantized after frequency conversion. As a result, the generated code amount differs for each reproduction unit (field, frame) or for each predetermined area in the image. And when performing intra-image compression, it is necessary to maintain the image quality only with that image,
Normally, a larger amount of code is allocated than in inter-image compression. For this reason, when moving image encoding is performed according to the MPEG standard, the generated code amount changes for each reproduction unit (field, frame). In practice,
The time management information is added to the packet header of the first packet in each playback unit (video frame, audio frame). Note that the reproduction unit does not always match the pack. For example, FIGS. 7 to 9 show a case where a plurality of packs constitute a reproduction unit.

Here, the data structure of the MPEG standard will be described. According to the MPEG standard, the data amount of a packet depends on a transmission medium and an application, and may be fixed length or variable length for each packet. MPE
In G1 and MPEG2-PS, a pack layer is provided above a packet, and a pack obtained by bundling a plurality of packets is usually handled as a constituent unit. Information for time reference reference for synchronous reproduction is added to the pack header of the pack. The pack, packet, and sector may be 1 pack = 1 packet = 1 sector as shown in FIG. 7, 1 pack = 1 packet as shown in FIG. 8, and 1 pack = 1 packet as shown in FIG. A pack may be set to a plurality of packets.
FIG. 10 shows a data structure in the case of 1 pack = 1 packet = 1 sector (= 2048 B).

[0049] The invention of claim 29 is the code recording apparatus according to claim 19, 22 or 26, wherein the arbitrary information is audio information. The invention according to claim 30 is the invention according to claim 19, 22 or 26, wherein the arbitrary information is audio information, and the other information includes audio information different from the arbitrary information. Code recording device. An invention according to claim 31 is the code recording device according to claim 26, wherein the arbitrary information is audio information and / or subtitle information. The invention according to claim 32 is the code recording apparatus according to claim 26, wherein the arbitrary information is audio information and subtitle information, and the other information includes audio information different from the arbitrary information. is there.

A thirty-third aspect of the present invention is the code recording apparatus according to any one of the eighteenth to thirty-second aspects, wherein the code sequence of the moving image information is a variable bit rate code sequence. The variable bit rate is, for example, GO in MPEG.
The generated code amount for each P (Group of Picture) fluctuates greatly, and refers to the case where the data transfer rate fluctuates with time.

According to a thirty-fourth aspect of the present invention, there is provided a code for multiplexing a code sequence of arbitrary information having the same time as the certain information with a packet sequence formed by packetizing a code sequence of certain information and recording the same on a recording medium. In the recording apparatus, when an offset position with respect to the certain information is designated with respect to the arbitrary information, position information indicating a position of the arbitrary information in a code string is an offset-considered position in consideration of the offset position. A conversion unit for converting the information into information, and packetizing the code string of the arbitrary information; and, when the target packet includes the head of the access unit at the time of packetization, offset consideration position information of the head access unit in the target packet. Packetizing means for generating time management information based on the packet and adding the time management information as header information of the target packet; Calculating means for obtaining a time-preceding relationship of a packet regardless of the type of information, based on the time-management information being added and the time-management information added to the packet of the arbitrary information by the packetizing means; And a multiplexing means for connecting each packet in chronological order with reference to FIG.

The invention according to claim 35 is a code for multiplexing a code sequence of arbitrary information having the same time as that of a certain information on a packet sequence formed by packetizing a code sequence of a certain information and recording the same on a recording medium. In the recording apparatus, when an offset position with respect to the certain information is designated with respect to the arbitrary information, position information indicating a position of the arbitrary information in a code string is an offset-considered position in consideration of the offset position. A conversion unit for converting the information into information, and packetizing the code string of the arbitrary information; and, when the target packet includes the head of the access unit at the time of packetization, offset consideration position information of the head access unit in the target packet. Packetizing means for generating time management information based on the packet and adding the time management information as header information of the target packet; The time management information added to the packet of the arbitrary information by the packetizing means is compared between different types of information, and packets having the same time management information are extracted as a simultaneous packet set. Extracting means, and multiplexing means for connecting the packets constituting the simultaneous packet set, and for connecting the simultaneous packet sets in time order based on the time management information,
Is a code recording device.

According to a thirty-sixth aspect of the present invention, there is provided a code for multiplexing a code sequence of arbitrary information having the same time as the certain information with a packet sequence obtained by packetizing a code sequence of certain information and recording the same on a recording medium. In the recording apparatus, when an offset position with respect to the certain information is designated with respect to the arbitrary information, position information indicating a position of the arbitrary information in a code string is an offset-considered position in consideration of the offset position. A conversion unit for converting the information into information, and packetizing the code string of the arbitrary information; and, when the target packet includes the head of the access unit at the time of packetization, offset consideration position information of the head access unit in the target packet. Packetizing means for generating time management information based on the packet and adding the time management information as header information of the target packet; The time management information added to the packet of the arbitrary information by the packetizing means is compared between different types of information, and packets having the same time management information are extracted as a simultaneous packet set. Extracting means, calculating means for obtaining a temporal predecessor-relationship based on the time management information regarding the simultaneous packet set extracted by the extracting means, and connecting the packets constituting the simultaneous packet set and referring to the precedent-rear relation And a multiplexing means for connecting each simultaneous packet set in time order.

According to a thirty-seventh aspect of the present invention, a packet sequence formed by packetizing a code sequence of arbitrary information into a packet sequence formed by packetizing a code sequence of certain information is multiplexed. A recording device for recording on a recording medium, when an offset position with respect to the certain information is specified for the arbitrary information, the time management information added to the packet of the arbitrary information is replaced with the offset position. Conversion means for converting the information into offset-considered time management information taking into account the type of information, based on the time management information added to the packet of the certain information and the offset-considered time management information converted by the conversion means. And a multiplexing means for chronologically linking the packets by referring to the preceding and following relations. A recording apparatus.

According to a thirty-eighth aspect of the present invention, a packet sequence formed by packetizing a code sequence of arbitrary information into a packet sequence formed by packetizing a code sequence of certain information is multiplexed. A recording device for recording on a recording medium, when an offset position with respect to the certain information is specified for the arbitrary information, the time management information added to the packet of the arbitrary information is replaced with the offset position. Means for converting into the offset-considered time management information considering the time, and comparing the time-consideration time management information added to the packet of the certain information with the offset-considered time management information converted by the conversion means, and calculating the time at the same time. Extracting means for extracting a packet having management information as a simultaneous packet set; connecting the packets constituting the simultaneous packet set with each other; And multiplexing means for each simultaneous packet assembly connecting time order based on the management information is a code recording apparatus having a.

In a thirty-ninth aspect of the present invention, a packet sequence formed by packetizing a code sequence of a certain information and a packet sequence formed by packetizing a code sequence of an arbitrary information having the same time as the certain information is multiplexed. A recording device for recording on a recording medium, when an offset position with respect to the certain information is specified for the arbitrary information, the time management information added to the packet of the arbitrary information is replaced with the offset position. Means for converting into the offset-considered time management information considering the time, and comparing the time-consideration time management information added to the packet of the certain information with the offset-considered time management information converted by the conversion means, and calculating the time at the same time. Extracting means for extracting a packet having management information as a simultaneous packet set; and the time management information for the simultaneous packet set extracted by the extracting means. And a multiplexing means for connecting the packets constituting the simultaneous packet set and for connecting the simultaneous packet sets in chronological order by referring to the preceding and succeeding relation. It is.

According to a fortieth aspect of the present invention, in any one of the thirty-fifth, thirty-sixth, thirty-eighth, and thirty-ninth aspects, the extracting means cuts off a predetermined number of lower-order bits of the time management information. A code recording apparatus for performing the extraction by referring to data of a predetermined number of upper bits. The invention of Claim 41 is Claim 35, Claim 36, Claim 38, Claim 39
In any one of the above, the extraction means is a code recording device that performs the extraction by rounding a predetermined number of lower-order bits of the time management information and referring to data of a predetermined upper-order number of bits.

According to a twenty-second aspect of the present invention, in any one of the thirty-fourth, thirty-six, thirty-seventh, and thirty-ninth aspects, the time management information includes decoding time information indicating a decoding time of an access unit and reproduction time. The code recording device is playback time information indicating a time, and the calculating means obtains a temporal predecessor-relationship based on either the decoding time information or the playback time information. The invention of Claim 43 is Claim 35,
40. The time management information according to claim 38, wherein the time management information is decoding time information indicating a decoding time of an access unit and reproduction time information indicating a reproduction time, and the multiplexing means is the decoding time information or the reproduction time information. Is a code recording device that links each set of simultaneous packets based on any one of the above.

According to a forty-fourth aspect, in the code recording apparatus according to any one of the thirty-fourth to forty-first aspects, the certain information includes at least moving picture information, and the arbitrary information is audio information. It is. The invention according to claim 45 is the invention according to claim 4.
42. The code recording apparatus according to claim 43, wherein the certain information includes at least moving image information, and the arbitrary information is audio information.

The invention of claim 46 is the invention according to claim 42 or claim 43, wherein the decoding time information is a DTS (Decoding Time Stamp) of the MPEG1 or MPEG2 standard, and the reproduction time information is a DTS (Decoding TimeStamp) of the MPEG1 or MPEG2 standard. This is a code recording device that is a PTS (Presentation Time Stamp).

According to a forty-seventh aspect of the present invention, the certain information is moving picture information, audio information and subtitle information, and the arbitrary information is another audio information. Is a code recording device. The invention of claim 48 is the code according to claim 42 or 43, wherein the certain information is moving image information, audio information, and caption information, and the arbitrary information is another audio information. It is a recording device.

The invention according to claim 49 is based on any one of claims 44 to 48, wherein the data amount of each access unit of the code string of the moving picture information is allowed to be different for each access unit. Where the code recording device is. An invention according to claim 50 is the code recording apparatus according to any one of claims 44 to 48, wherein the code sequence of the moving image information is a code sequence of a variable bit rate.

A fifty-first aspect of the present invention is a code multiplexing method for multiplexing a plurality of types of code sequences having the same time characteristic, wherein each code sequence is based on position information indicating a position in each code sequence. Is a code multiplexing method that obtains a temporal predecessor-relationship that is not restricted by a code sequence for the access unit of (i), divides each of the code sequences into packets, and multiplexes each packet in time order based on the predecessor-relationship.

The invention according to claim 52 is a code multiplexing method for multiplexing a plurality of types of information code strings having the same time characteristic, wherein each code string is based on position information indicating a position in each code string. For each of the access units, a temporal predecessor-relationship that is not restricted to the code string is obtained, and each of the code strings is packetized. When the target packet includes the head of the access unit during packetization, the first access in the target packet is performed. This is a code multiplexing method in which time management information is generated based on position information of a unit, added as header information of the target packet, and each packet is connected in time order based on the preceding and following relationship.

The invention according to claim 53 is a code multiplexing method for multiplexing a plurality of types of information code strings having the same time characteristic, wherein each of the code strings is packetized, and a target packet is accessed at the time of packetization. When the head of the unit is included, time management information is generated based on the position information of the head access unit in the target packet and added as header information of the target packet, and the generated packet is added to the time management information. This is a code multiplexing method for obtaining a temporal predecessor-relationship based on the type based on the type, and connecting the packets in time order with reference to the predecessor-relationship.

A fifty-fourth aspect of the present invention is a code multiplexing method for multiplexing a plurality of types of code strings of information having the same time, wherein each of the code strings is packetized, and a target packet is accessed at the time of packetization. When the head of the unit is included, time management information is generated based on the position information of the head access unit in the target packet and added as header information of the target packet, and the time management information of the generated packet is classified Extracting the packets having the time management information at the same time by comparing between the different pieces of information as a simultaneous packet set, obtaining a temporal predecessor-relationship based on the time management information with respect to the extracted simultaneous packet set, A code for connecting the packets constituting the set and for connecting the simultaneous packet sets in chronological order with reference to the preceding and following relations, It is a heavy way.

A fifty-fifth aspect of the present invention is a code multiplexing method for multiplexing a plurality of types of code sequences having the same time, wherein an arbitrary information in the plurality of types of information is offset with respect to other information. Specify, and convert the position information indicating the position in the code string of the arbitrary information into offset-considered position information considering the specified offset position,
Based on the offset-considered position information in the code string of the arbitrary information and the position information indicating the position of the other information in the code string, a temporal sequence of the access units of each code string that is not restricted to the code string is determined. , A code multiplexing method of packetizing each of the code strings and multiplexing each of the packets in time order based on the preceding and following relations.

The invention according to claim 56 is a code multiplexing method for multiplexing a code string of a plurality of types of information having the same time, wherein an arbitrary information in the plurality of types of information is offset with respect to other information. Specify, and convert the position information indicating the position in the code string of the arbitrary information into offset-considered position information considering the specified offset position,
Based on the offset-considered position information in the code string of the arbitrary information and the position information indicating the position of the other information in the code string, a temporal sequence of the access units of each code string that is not restricted to the code string is determined. When each of the code strings is packetized, and when the target packet includes the head of the access unit at the time of packetization, the time management information is based on the position information or offset-considered position information of the head access unit in the target packet. Is generated and added as header information of the target packet, and each packet is connected in chronological order based on the preceding and following relations.

A fifty-seventh aspect of the present invention is a code multiplexing method for multiplexing code strings of a plurality of types of information having the same time, wherein an arbitrary information in the plurality of types of information is offset with respect to other information. Specify, and convert the position information indicating the position in the code string of the arbitrary information into offset-considered position information considering the specified offset position,
Each of the code strings is packetized, and when the target packet includes the head of the access unit at the time of packetization, time management information is determined based on the position information or offset-considered position information of the head access unit in the target packet. Generated and added as header information of the target packet, a temporally preceding and following relationship is determined for the generated packet regardless of the type based on the time management information, and the respective packets are connected in chronological order with reference to the preceding and following relationship. This is a code multiplexing method.

A fifty-eighth invention is a code multiplexing method for multiplexing a plurality of types of information code streams having the same time, wherein an arbitrary information in the plurality of types of information is offset with respect to other information. Specify, and convert the position information indicating the position in the code string of the arbitrary information into offset-considered position information considering the specified offset position,
Each of the code strings is packetized, and when the target packet includes the head of the access unit at the time of packetization, time management information is determined based on the position information or offset-considered position information of the head access unit in the target packet. Generated and added as header information of the target packet, comparing the time management information of the generated packet between the different types of information, extracting packets having the same time management information as a simultaneous packet set, A time-preceding relationship is determined based on the time management information with respect to the extracted simultaneous packet set, and the packets constituting the simultaneous packet set are connected, and the simultaneous packet sets are connected in time order with reference to the preceding-and-backward relationship. It is a multiplex method.

The invention according to claim 59 is a code multiplexing method for multiplexing a code sequence of arbitrary information having the same time characteristic as a certain information into a packet sequence of certain information. The position of the arbitrary information in the code sequence is converted into offset-considered position information in consideration of the specified offset position, and the code sequence of the arbitrary information is converted into a packet. If the target packet includes the head of the access unit during packetization, time management information is generated based on the offset-considered position information of the head access unit in the target packet, and is used as header information of the target packet. Time management information added to the certain information packet and the time management information added to the arbitrary information packet. Zui seek time antecedent relationship packets regardless of the type of information, it is connected by referring to the destination after the relationship in order to each packet time, a code multiplexing method.

The invention according to claim 60 is a code multiplexing method for multiplexing a code string of arbitrary information having the same time property as a certain information into a packet string of certain information. The position of the arbitrary information in the code sequence is converted into offset-considered position information in consideration of the specified offset position, and the code sequence of the arbitrary information is converted into a packet. If the target packet includes the head of the access unit during packetization, time management information is generated based on the offset-considered position information of the head access unit in the target packet, and is used as header information of the target packet. Time management information added to the certain information packet and the time management information added to the arbitrary information packet. By comparing packets of different types, packets having the same time management information at the same time are extracted as a simultaneous packet set, the packets constituting the simultaneous packet set are connected, and each of the simultaneous packet sets is extracted based on the time management information. Are multiplexed in time sequence.

The invention according to claim 61 is a code multiplexing method for multiplexing a code string of arbitrary information having the same time property as a certain information into a packet string of certain information, wherein The position of the arbitrary information in the code sequence is converted into offset-considered position information in consideration of the specified offset position, and the code sequence of the arbitrary information is converted into a packet. If the target packet includes the head of the access unit during packetization, time management information is generated based on the offset-considered position information of the head access unit in the target packet, and is used as header information of the target packet. Time management information added to the certain information packet and the time management information added to the arbitrary information packet. Extracting packets having the same time management information at the same time by comparing different types of information as a simultaneous packet set, obtaining a temporal predecessor-relationship based on the time management information for the extracted simultaneous packet set, This is a code multiplexing method in which packets forming a set are connected and each simultaneous packet set is connected in time sequence with reference to the preceding and following relations.

An invention according to claim 62 is a code multiplexing method for multiplexing a packet sequence of arbitrary information having the same time characteristic as a certain information in a packet sequence of certain information. The time management information added to the packet of the arbitrary information is converted into the offset-considered time management information in consideration of the specified offset position, and is added to the packet of the certain information. Based on the time management information being converted and the converted offset-considered time management information, to determine the temporal pre / post relationship of the packets irrespective of the type of information, and refer to the pre / post relationship to link each packet in time order, code multiplexing. Is the way.

The invention according to claim 63 is a code multiplexing method for multiplexing a packet sequence of arbitrary information having the same time as that of a certain information into a packet sequence of a certain information. The time management information added to the packet of the arbitrary information is converted into the offset-considered time management information in consideration of the specified offset position, and is added to the packet of the certain information. Comparing the time management information with the converted offset-considered time management information and extracting the packets having the time management information at the same time as a simultaneous packet group, connecting the packets constituting the simultaneous packet group, and This is a code multiplexing method in which each simultaneous packet group is connected in time order based on management information.

An invention according to claim 64 is a code multiplexing method for multiplexing a packet sequence of arbitrary information having the same time as that of a certain information in a packet sequence of the certain information. The time management information added to the packet of the arbitrary information is converted into the offset-considered time management information in consideration of the specified offset position, and is added to the packet of the certain information. Comparing the time management information with the converted offset-considered time management information to extract packets having time management information at the same time as a simultaneous packet set, and perform time processing on the extracted simultaneous packet set based on the time management information. Find the target-relationship relationship, connect the packets that make up the simultaneous packet set, and reference each The coupling time order, a code multiplexing method.

The invention of claim 65 is the invention of claim 54, claim 58, claim 60, claim 61, claim 63, claim 6.
4. The code multiplexing method according to any one of 4), wherein the predetermined number of lower-order bits of the time management information is truncated and the extraction is performed with reference to data of a predetermined upper-order number of bits. Claim 66
The invention according to any one of claims 54, 58, 60, 61, 63, and 64, wherein a predetermined number of lower bits of the time management information are rounded to a predetermined upper number. Is a code multiplexing method in which the extraction is performed with reference to the data of the bits.

[0078]

FIG. 1 is a block diagram showing an embodiment of the present invention. FIG. 2 shows an MPEG system encoder for the stream output from the video encoder 210.
The data structure of the pack header / system header / packet added in 410 and the data structure in the packet, or
The data structure of a pack header, a system header, and a packet added to the stream output from the audio encoder 230 by the MPEG system encoder 410 and the data structure in the packet are shown.

The video data input to the code recording apparatus of FIG.
After being compression-encoded according to a known video compression method using M220, it is input to the MPEG system encoder 410. The audio data input to the code recording device of FIG.
Compressed and encoded according to a well-known audio compression method using a RAM 240 in an MPEG system encoder.
Entered into 410.

Each of the video encoder 210 and the audio encoder 230 performs compression encoding / transfer processing at predetermined fixed time intervals. In other words, compression coding / transfer processing (block coding) for each unique frame unit
Is performed.

For example, in the video encoder 210, the compression encoding process is performed in real time according to the order of recording on the recording medium 90 for each video frame (1/30 second), and the encoded compressed video data is converted into video frame units. Is transferred to the MPEG system encoder 410. Since the amount of compressed data of each video frame is different, 1/30
The amount of data transferred every second also varies.

In the audio encoder 230, 1
The compression encoding process is performed intermittently for each audio frame, and the encoded compressed audio data is transferred to the MPEG system encoder 410 in audio frame units. Since the amount of compressed data of each audio frame is substantially constant depending on the encoding format (average transfer rate), the amount of data transferred for each audio frame period is also substantially constant. However, when the encoding format is changed, the compressed data amount of each audio frame naturally changes.

The unit of the compression encoding / transfer processing time in the audio encoder 12 is the same as the time of one video frame (1/30 second) which is the unit of the compression encoding / transfer processing time in the video encoder 11. However, different times are generally used.

The MPEG system encoder 410 performs the MPEG system encoding on the compressed video data and the compressed audio data stored in the RAM 41a during the predetermined time τ. That is, it is packed and packetized. Here, considering the format method of the recording medium 90, 2K
Byte packing is performed.

The data in units of frames processed by the audio encoder 230 and the video encoder 210 are collectively stored in the RAM 410a for each frame unit. The data amount of the video frames is I, P,
It differs greatly from frame to frame because of the difference in encoding methods such as B pictures and the fact that the number of bits allocated in GOP units also differs in the case of a variable bit rate. In contrast, the amount of audio frame data is
Unless the encoding format (average transfer rate) is changed, it is almost constant. Therefore, if multiplexing is performed as it is, there is a problem that the memory capacity of the AV rate control RAM in the decoder is increased as described above.

For this reason, video and audio compressed data once stored in the RAM 410a are
MPEG system encoding is performed by adding time management information based on the value from the counter that measures the 0 kHz clock. Specifically, DTS and PTS are added as time management information at the time of packing and packetization. In the present invention, at this time, time management information of each compressed data of video and audio is corresponded and multiplexed side by side. If it is desired to start multiplexing audio from a desired position in the video, the desired position is given as an offset, and the DTS and PTS of the audio are shifted by the offset.

This is output to the formatter 43. That is, the MPEG system encoder 410 functions as an arithmetic unit, a packetizing unit, a multiplexing unit, and an extracting unit described in the claims except for the one having the converting unit. In this way, the parameters of the packet header such as PTS and DTS are determined, and a multiplexed AV bit stream arranged according to the functions of the calculating means, the packetizing means, the multiplexing means, and the extracting means is obtained. Here, the PTS and DTS are added to the packet header of a packet in which the head of an audio or video access unit (= reproduction unit: video frame / audio frame) exists.

In the above-described MPEG system encoding, the standard of the recording medium 90 is considered. That is, the recording medium
Numeral 90 is divided into sectors in consideration of file access, and one sector usually includes two sectors as shown in FIG.
It consists of KByte data and additional data such as error correction code and address data. The encoding process is performed by the MPEG system encoder 410 so as to conform to the sector structure of the recording medium 90. That is, the encoding process is performed by the MPEG system encoder 410 so that the pack packet length of the multiplexed AV bit stream conforms to the above-mentioned sector length. For example, the following formatter
After the processing in 43, the MPEG system encoder 410 performs MPEG system encoding so that one pack becomes 2 Kbytes and one pack is recorded in one sector.

The multiplexed AV bit stream encoded by the MPEG system encoder 410 is sent to the formatter 43.

The formatter 43 stores the multiplexed AV bit stream input from the MPEG system encoder 41 in the RAM 42 for each sector. The storage amount is a recording block unit amount (eg, 32 sectors / 64 sectors / 12
Each time the data reaches 8 sectors), the multiplexed AV bit stream is read from the RAM 42, added with an error correction code and the like, and sent to the write circuit 50. As a result, the multiplexed AV bit stream to which the error correction code has been added is recorded on the recording medium 90.

The formatter 43 performs signal processing for a recording format. This signal processing performs logical formatting and physical formatting so as to conform to the standard of the recording medium. In the preceding MPEG system format, the processing according to the physical format in the formatter 43 is performed in advance as described above. The recording medium is divided into sector units for file access, and usually includes 2 Kbytes of data, additional data such as address data, and data such as an error correction code. For this reason, the pack packet length is also formatted according to the sector length. For example, when packed, it becomes 2 Kbytes, and one pack is applied to one sector. By doing so, consistency with the signal processing of the recording format is obtained.

In the RAM 42 attached to the formatter 43, M
The PEG system stream is accumulated in sector units, and writing is continued until a recording block is obtained. When the data is stored up to the unit of the recording block (usually a unit of 32, 64, 128 sectors or the like, which differs depending on the standard of the recording medium), the formatter 43 adds a signal (error correction code or the like) to the writing circuit. Writing is performed on the recording medium 90 through 50.

When the encoding speed and the writing speed do not match, such block writing is indispensable. In particular, in a stream corresponding to a variable bit rate, the average encoding speed changes every moment, so that processing is performed in block units while adjusting the speed using the buffer memory 42.

In the case of the fixed bit rate, since the average coding rate is substantially constant (however, there is considerable fluctuation in microscopic terms), if the coding rate and the writing rate are matched,
Continuous writing can be realized. Even in such a case, the buffer memory 42 is required, although having a relatively small capacity, in order to absorb microscopic fluctuations.

In the above description, an example is described in which the data is read out from the RAM 42 and then formatted into a recording format by the formatter 43. However, before the data is stored in the RAM 42, an error correction code or the like is added by the formatter 43 and the error correction code is added. The bit stream to which a code or the like has been added may be stored in the RAM 42 for each sector. In that case, RAM
It is necessary to increase the capacity of 42.

The specific configuration of the writing circuit 50 is determined according to the type of the recording medium 90. For example, when the recording medium 90 is a high-density magneto-optical disk, a head capable of recording data on the high-density magneto-optical disk and its driving circuit are used. As the writing circuit 50 corresponding to the type of the recording medium 90, a writing circuit corresponding to a known recording medium can be used. For example, as the recording medium 90,
Optical disks such as CD, CD-ROM, LD, LD-ROM, magneto-optical disks such as MO and MD, hard disks,
Various recording media such as a magnetic disk such as a floppy disk, a magnetic tape, an optical tape, an IC memory, and a high-recording-density DVD are applicable.

FIG. 1 shows the case where the types of input data are compressed video data and compressed audio data, but the types of data are not limited to these, and may be character information such as subtitles. . That is, any information that requires simultaneous reproduction of the corresponding access units, in other words, information having the same time characteristic may be used.

In the MPEG system encoder 410, the compressed data input from the video encoder 210 and the audio encoder 230 and stored in the RAM 410a are
Time management information (PTS / DTS) is added and packed.
Perform packetization. Video with time management information
Each packet data of the audio is compared with the time management information, and the time management information is sorted in the early order (video or audio).
Are sorted and packed. Alternatively, when time management information is added to each piece of video / audio data, there is a method of comparing time management information to be added to each piece of data, and rearranging and packing the data in the same manner. At this time, the sector to which the time management information is not added is regarded as a sector having time information preceding in time and having the same time information as the sector detected first. Then, the packed data is sent to the formatter 43. The subsequent processing is as described above.

[0099] Although the above describes real-time processing, it is necessary to packetize each encoded stream and store data to which time information is added, and to perform multiplexing processing on the stored data. Can also. That is, non-real-time processing can be performed.

Next, a method of rearranging according to time management information will be described. As shown in FIG. 3, in a video bit stream and an audio bit stream, generally,
The time unit of each one frame is different. Therefore, the timestamps of video and audio packets to be associated with each other often do not coincide.

In this case, as shown in FIG. 5, video and audio packets may be arranged in time order.
When the time stamp indicates the same time, one of the video and the audio is set ahead. For example, if the times are the same, the audio always precedes the video. In FIG. 5, t
.., 0, t1, t2,... Are times indicated by the time stamps added to the video and audio packets, and the younger the number, the earlier the time.

As shown in FIG. 4, the upper bits (upper 24 bits in FIG. 4) of the time stamp added to the video and audio packets are extracted, and the extracted upper 24 bits are searched for. You may comprise so that it may be set as a set of time and multiplexed. As a method of extracting the upper bits serving as the search index, there are a method of cutting off lower bits (lower 9 bits in FIG. 4) and a method of rounding lower bits (lower 9 bits in FIG. 4). Thus, the degree of ignoring the lower bits becomes the time resolution when multiplexing the video and audio packets.

FIG. 6 shows a case where a video bit stream, an audio bit stream, and a bit stream of caption data are arranged in the order of time management information and multiplexed into a multiplexed bit stream. In FIG. 6, t0, t1, t2,... Are the times indicated by the time stamps added to the video, audio, and subtitle packets, and the younger the number, the earlier the time.

In the above example, a case has been described in which a plurality of types of information such as video / audio / subtitles are all multiplexed using the same time as the reference of the start time. It is also possible to multiplex such that one or more types of information start at an arbitrary time.

Hereinafter, such an example will be described with reference to FIGS. That is, the MPEG system encoder 410 shown in FIG. 1 is provided with a converting means, the functions of an arithmetic means, a function of a packetizing means, a function of a multiplexing means, a function of an extracting means, and a converting means. A configuration example having the above function will be described. Such a configuration is useful, for example, when it is desired to mix another audio from a desired time into a multiplexed bit stream in which video and audio are multiplexed.

FIG. 16 shows a case where a packed and packetized audio bit stream is multiplexed from a position in the middle of the packed and packetized video bit stream. FIG. 17 shows a case where a packed / packetized bit stream of character information is multiplexed from the middle of a multiplexed stream of a packed / packetized video bit stream. Here, 1 pack =
The case of one packet will be described.

As shown in FIG. 16, an audio bit stream whose head is A1 is converted into a video bit stream V.
When inserting between 1 and V2, first, the time stamp of V3 is read, and the time stamp of A1 is set to the same time as the time stamp of V3. That is, the offset is given so that the time stamp of A1 is shifted by the time stamp of V3. As a result, A1 and V3 form a packet set having the same time stamp.
After A2, since the time stamp of V3 is given as an offset and shifted by the time stamp of V3, similarly, the shifted time stamp is compared with the time stamp of the video packet and multiplexed similarly. .

If there is no time stamp in the packet of V3, the immediately preceding time stamp in the video bit stream is read, and the immediately preceding time stamp is given to the audio bit stream as an offset. Processing is performed in the same manner as described above. Alternatively, the insertion position may be moved to the video packet preceding immediately before V3.

The above multiplexing method is used when both the video bit stream and the audio bit stream multiplexed from the middle of the video bit stream are already packetized (in the case of non-real-time processing). In the case where the audio bit stream is not packetized, that is, in the case of real-time processing in which the compressed audio data output from the encoder is multiplexed from the middle of the already-packetized video bit stream, the same technique is applied. Can be. Also, if both video and audio are not packetized,
Similar techniques can be applied.

As shown in FIG. 17, even when character information is multiplexed in the middle of a multiplexed stream of video and audio, the same processing can be performed regardless of non-real-time processing or real-time processing. The same applies to the case where different audio information is multiplexed in the middle of a multiplexed stream of video and audio. 16 and FIG.
7, the packet is arranged in the order of the time indicated by the time stamp. As a result, a case in which packets of any one type of information continue occurs naturally.

[0111]

According to the present invention, the packets of compressed video data and compressed audio data at substantially the same time are multiplexed so as to be close to each other in the multiplexed stream, so that they are stored almost simultaneously in the AV rate control RAM of the decoder. And the capacity of the RAM can be reduced. Further, this effect can be achieved with a simple configuration. Further, the above-described multiplexed stream can be configured in real time. In addition, the above-described multiplexed stream can be configured in non-real time.

Further, according to the present invention, audio and subtitles can be multiplexed and recorded from an arbitrary position in a moving image. Also, subtitles and other audio can be multiplexed and recorded from an arbitrary position of the multiplexed data of the moving image and the audio. Also, it is possible to multiplex and record the compression-encoded data of the moving image and the sound, and to multiplex and record a desired subtitle and another sound from an arbitrary position. In other words, different information can be multiplexed and recorded from an arbitrary position of a plurality of types of information requiring the same time.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of the apparatus of the present invention.

FIG. 2 is a video or audio bit stream in which time stamps PTS and DTS are added to a packet header of a first packet.

FIG. 3 is an explanatory diagram showing that since time units of one frame of a video bit stream and an audio bit stream are different, time stamps added corresponding to respective frames are slightly different from each other.

FIG. 4 is an explanatory diagram showing a case where upper bits of each time stamp of a video pack and an audio pack are extracted and a time stamp indicating substantially the same time is searched.

FIG. 5 is an explanatory diagram showing an example in which video packs and audio packs are arranged in chronological order to form a multiplexed bit stream.

FIG. 6 is an explanatory diagram showing an example in which video packs, audio packs, and packs of subtitle information are arranged in order of time to form a multiplexed bit stream.

FIG. 7 is an explanatory diagram of a data configuration of one pack, one packet, and one sector.

FIG. 8 is an explanatory diagram of a data structure of one pack and one packet.

FIG. 9 is an explanatory diagram of a data structure of one pack and a plurality of packets.

FIG. 10 is an explanatory diagram of a data structure of one pack, one packet, and one sector of MPEG.

FIG. 11 is a block diagram showing a configuration example of a DVD decoder.

FIG. 12 is a block diagram showing a configuration example of a decoder for decoding a video-CD in which a bit stream of the MPEG standard is recorded.

FIG. 13 is a block diagram showing a configuration example of a decoder for decoding a DVD on which a bit stream of the MPEG2 standard is recorded.

FIG. 14 is an explanatory diagram showing a configuration example of compressed video data at a fixed bit rate.

FIG. 15 is an explanatory view showing a configuration example of compressed video data of a variable bit rate.

FIG. 16 is an explanatory diagram showing an example of multiplexing an audio bit stream from the middle of a video bit stream.

FIG. 17 is an explanatory diagram showing an example in which a bit stream of character information is multiplexed from the middle of a multiplexed bit stream of video and audio.

[Explanation of symbols]

 410 MPEG system encoder 410a RAM

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Minoru Taoka, Inventor Sanyo Electric Co., Ltd. 2-5-2-5 Keihanhondori, Moriguchi-shi, Osaka

Claims (66)

[Claims] 1. A code recording apparatus for multiplexing a plurality of types of information code strings having the same time and recording the same on a recording medium, wherein each code is based on position information indicating a position in each of the code strings. Code means comprising: a calculating means for obtaining a time-preceding relationship that is not restricted to a code string with respect to a column access unit; and a multiplexing means for packetizing each code string and multiplexing each packet in time order based on the preceding-and-significant relation. Recording device. 2. The multiplexing means according to claim 1, wherein, when the packet includes a head of an access unit, the multiplexing means generates time management information based on position information of a head access unit in the packet. A code recording device that is added as packet header information. 3. A code recording device for multiplexing a plurality of types of code strings of information having the same time and recording the same on a recording medium, wherein each code is based on position information indicating a position in each of the code strings. Calculating means for determining a temporal sequence of the access units of the sequence, which is not restricted to the code sequence; and packetizing each of the code sequences, and when the target packet includes the head of the access unit at the time of packetization, the Packetizing means for generating time management information based on the position information of the first access unit and adding the time management information as header information of the target packet, and multiplexing means for connecting each packet in time order based on the preceding and following relationship. Code recording device. 4. A code recording apparatus for multiplexing a plurality of types of information code sequences having the same time and recording the same on a recording medium, wherein each of the code sequences is packetized and a target packet is A packetization unit that generates time management information based on the position information of the head access unit in the target packet and adds the time management information as header information of the target packet when including the head of the access unit; A code recording apparatus comprising: a calculating unit that obtains a time-preceding relationship irrespective of the type of the generated packet based on the time management information; . 5. A code recording apparatus for multiplexing a plurality of types of information code sequences having the same time and recording the same on a recording medium, wherein each of the code sequences is packetized, and a target packet is used for packetization. A packetization unit that generates time management information based on the position information of the head access unit in the target packet and adds the time management information as header information of the target packet when including the head of the access unit; Extracting means for comparing the time management information of the generated packets between the different types of information and extracting packets having the time management information at the same time as a simultaneous packet set; and connecting the packets constituting the simultaneous packet set. And multiplexing means for connecting each simultaneous packet set in time order based on the time management information. Location. 6. A code recording apparatus for multiplexing a plurality of types of information code strings having the same time and recording the same on a recording medium, wherein each of the code strings is packetized, and a target packet is used for packetization. A packetization unit that generates time management information based on the position information of the head access unit in the target packet and adds the time management information as header information of the target packet when including the head of the access unit; Extracting means for comparing the time management information of the generated packets between the different types of information and extracting the packets having the time management information at the same time as a simultaneous packet group; and regarding the simultaneous packet group extracted by the extracting means. Calculating means for obtaining a temporal pre / post relationship based on the time management information; and a packet forming the simultaneous packet set. A multiplexing means for connecting a reference to the simultaneous packet sets the destination after relationship in time order with linking, code recording apparatus having a. 7. A code recording apparatus for multiplexing a plurality of types of information code strings having the same time and recording the same on a recording medium, wherein an offset position with respect to other information with respect to arbitrary information in the plurality of types of information. Is specified, converting means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information in consideration of the offset position; and Calculating means for determining a temporally-preceding relationship that is not restricted to a code string with respect to an access unit of each code string based on offset-considered position information and position information indicating a position of the other information in the code string; Multiplexing means for packetizing and multiplexing each packet in chronological order based on the preceding and following relation. 8. The multiplexing means according to claim 7, wherein, when the packet includes a head of an access unit, the multiplexing means performs time management information based on position information or offset-considered position information of a head access unit in the packet. , And adds it as header information of the packet. 9. A code recording apparatus for multiplexing a plurality of types of information code streams having the same time and recording the same on a recording medium, wherein an arbitrary information in the plurality of types of information is offset from other information. Is specified, converting means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information in consideration of the offset position; and Calculating means for determining a temporally-preceding relationship that is not restricted to a code string with respect to an access unit of each code string based on offset-considered position information and position information indicating a position of the other information in the code string; In addition to packetization, if the target packet includes the head of an access unit at the time of packetization, the position information or the position information of the head access unit in the target packet is used. A packet recording unit that generates time management information based on the offset-considered position information and adds the time management information as header information of the target packet; and a multiplexing unit that links each packet in time order based on the preceding and following relationship. . 10. A code recording apparatus for multiplexing a plurality of types of information code streams having the same time and recording the same on a recording medium, wherein an arbitrary information in the plurality of types of information is offset with respect to other information. Is specified, a conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position, and packetizing each of the code strings. If the target packet includes the head of the access unit during packetization, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet, and is used as header information of the target packet. A packetizing unit to be added; and a packet generated by the packetizing unit, based on the time management information. A code recording apparatus, comprising: an arithmetic unit that obtains a temporal predecessor relation irrespective of a type; and a multiplexing unit that connects respective packets in time order with reference to the predecessor relation. 11. A code recording apparatus for multiplexing a plurality of types of information code strings having the same time and recording the same on a recording medium, wherein an arbitrary information in the plurality of types of information is offset from other information. Is specified, a conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position, and packetizing each of the code strings. If the target packet includes the head of the access unit during packetization, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet, and is used as header information of the target packet. Packetizing means for adding, and time management information of a packet generated by said packetizing means between the information of different types. Extracting means for extracting packets having the same time management information at the same time as a simultaneous packet set, connecting the packets making up the simultaneous packet set, and converting each simultaneous packet set based on the time management information to time. A code recording device comprising: 12. A code recording apparatus for multiplexing a plurality of types of information code streams having the same time and recording the same on a recording medium, wherein an arbitrary information in the plurality of types of information is offset with respect to other information. Is specified, a conversion means for converting position information indicating the position of the arbitrary information in the code string into offset-considered position information considering the offset position, and packetizing each of the code strings. If the target packet includes the head of the access unit during packetization, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet, and is used as header information of the target packet. Packetizing means for adding, and time management information of a packet generated by said packetizing means between the information of different types. Extracting means for extracting packets having the same time management information at the same time as a simultaneous packet set, and calculating means for obtaining a temporal pre / post relationship based on the time management information with respect to the simultaneous packet sets extracted by the extracting means And a multiplexing means for connecting the packets constituting the simultaneous packet set and connecting each simultaneous packet set in time order with reference to the preceding and following relations. 13. The time management information according to claim 5, wherein the extraction means cuts off a predetermined number of lower-order bits of the time management information and a predetermined number of upper-order bits of the time management information. A code recording device that performs the extraction with reference to the data of (1). 14. The method according to claim 5, wherein the extracting means rounds a predetermined number of lower bits of the time management information and a predetermined number of upper bits of the time management information. A code recording device that performs the extraction with reference to the data of (1). 15. The time management information according to claim 2, wherein the time management information is decoding time information indicating a decoding time of an access unit and a reproduction time indicating a reproduction time. A code recording device that is time information. 16. The time management information according to claim 4, wherein the time management information is decoding time information indicating a decoding time of an access unit and a reproduction time indicating a reproduction time. A code recording device, which is time information, wherein the arithmetic unit obtains a temporal predecessor-relationship based on either the decoding time information or the reproduction time information. 17. The time management information according to claim 5, wherein the time management information is decoding time information indicating a decoding time of an access unit and reproduction time information indicating a reproduction time. A code recording device, wherein each simultaneous packet set is linked based on either the decoding time information or the reproduction time information. 18. The code recording apparatus according to claim 1, wherein the plurality of types of information are moving image information and audio information. 19. The code recording apparatus according to claim 7, wherein the plurality of types of information are moving image information and audio information. 20. The code recording apparatus according to claim 15, wherein the plurality of types of information are moving image information and audio information. 21. The code sequence according to claim 18, wherein the code sequence of the plurality of types of information is a code sequence of compressed code data obtained by compressing moving image information and audio information in accordance with the MPEG1 or MPEG2 standard. A code recording device that is a frame and one audio frame. 22. The method according to claim 19, wherein the code sequence of the plurality of types of information is a code sequence of compression code data obtained by compressing moving image information and audio information in accordance with the MPEG1 or MPEG2 standard. A code recording device that is a frame and one audio frame. 23. The code sequence according to claim 20, wherein the code sequence of the plurality of types of information is a code sequence of compression code data obtained by compressing moving image information and audio information in accordance with the MPEG1 or MPEG2 standard. A code recording device that is a frame and one audio frame. 24. The method according to claim 23, wherein the decoding time information is a DTS (Decoding Time Stamp), and the reproduction time information is a PTS (Presentation Time Stamp).
amp), a code recording device. 25. The code recording apparatus according to claim 1, wherein the plurality of types of information are moving image information, audio information, and subtitle information. 26. The code recording apparatus according to claim 7, wherein the plurality of types of information are moving image information, audio information, and subtitle information. 27. The code recording apparatus according to claim 15, wherein the plurality of types of information are video information, audio information, and subtitle information. 28. The code recording apparatus according to claim 21, wherein a data amount of each access unit of the code sequence of the moving image information is allowed to be different for each access unit. . 29. The code recording apparatus according to claim 19, wherein the arbitrary information is audio information. 30. The code recording method according to claim 19, wherein the arbitrary information is audio information, and the other information includes audio information different from the arbitrary information. apparatus. 31. The code recording apparatus according to claim 26, wherein the arbitrary information is audio information and / or subtitle information. 32. The code recording apparatus according to claim 26, wherein the arbitrary information is audio information and subtitle information, and the other information includes audio information different from the arbitrary information. 33. The code recording apparatus according to claim 18, wherein the code sequence of the moving image information is a variable bit rate code sequence. 34. A code recording apparatus which multiplexes a code string of arbitrary information having the same time characteristic as the certain information with a packet string formed by packetizing a code string of certain information, and records the multiplexed data on a recording medium. When an offset position with respect to the certain information is specified for the arbitrary information, conversion for converting position information indicating a position of the arbitrary information in a code string into offset-considered position information considering the offset position. Means for packetizing the code string of the arbitrary information and time management based on the offset-considered position information of the leading access unit in the target packet if the target packet includes the head of the access unit in the packetization. Packetizing means for generating information and adding the information as header information of the target packet; and a time tube added to the packet of the certain information. Calculating means for determining a temporal pre / post relationship of a packet irrespective of the type of information based on logical information and time management information added to the packet of the arbitrary information by the packetizing means, referring to the pre / post relationship Multiplexing means for connecting each packet in time order; 35. A code recording apparatus for multiplexing a code string of arbitrary information having the same time characteristic as a certain information with a packet string formed by packetizing a code string of a certain information, and recording the multiplexed code string on a recording medium. When an offset position with respect to the certain information is specified for the arbitrary information, conversion for converting position information indicating a position of the arbitrary information in a code string into offset-considered position information considering the offset position. Means for packetizing the code string of the arbitrary information and time management based on the offset-considered position information of the leading access unit in the target packet if the target packet includes the head of the access unit in the packetization. Packetizing means for generating information and adding the information as header information of the target packet; and a time tube added to the packet of the certain information. Extracting means for comparing time management information added to a packet of the arbitrary information by the packetizing means with information of different types and extracting packets having time management information at the same time as a simultaneous packet set; And a multiplexing means for connecting the packets constituting the simultaneous packet set and connecting each simultaneous packet set in time order based on the time management information. 36. A code recording apparatus for multiplexing a code sequence of arbitrary information having the same time characteristic as the certain information with a packet sequence formed by packetizing a code sequence of certain information, and recording the multiplexed code sequence on a recording medium. When an offset position with respect to the certain information is specified for the arbitrary information, conversion for converting position information indicating a position of the arbitrary information in a code string into offset-considered position information considering the offset position. Means for packetizing the code string of the arbitrary information and time management based on the offset-considered position information of the leading access unit in the target packet if the target packet includes the head of the access unit in the packetization. Packetizing means for generating information and adding the information as header information of the target packet; and a time tube added to the packet of the certain information. Extracting means for comparing time management information added to a packet of the arbitrary information by the packetizing means with information of different types and extracting packets having time management information at the same time as a simultaneous packet set; Calculating means for obtaining a temporal predecessor / relationship based on the time management information with respect to the simultaneous packet set extracted by the extracting means; and Multiplexing means for connecting the packet sets in time sequence; 37. A packet sequence formed by packetizing a code sequence of a certain information and a packet sequence formed by packetizing a code sequence of arbitrary information having the same time property as the certain information, and recording the packet sequence on a recording medium. An offset position for the certain information is specified with respect to the arbitrary information, the time management information added to the packet of the arbitrary information is taken into consideration by the offset considering the offset position. Conversion means for converting the time information into time management information; time of the packet regardless of the type of information based on the time management information added to the packet of the certain information and the offset-considered time management information converted by the conversion means. A code recording apparatus, comprising: an arithmetic unit for obtaining a target-relationship relationship; and a multiplexing unit for connecting each packet in chronological order with reference to the relationship. 38. A packet sequence formed by packetizing a code sequence of a certain information and a packet sequence formed by packetizing a code sequence of an arbitrary information having the same time characteristic as the certain information, and recording the resultant on a recording medium. An offset position for the certain information is specified with respect to the arbitrary information, the time management information added to the packet of the arbitrary information is taken into consideration by the offset considering the offset position. A conversion unit for converting the time management information into time management information; a packet having time management information at the same time by comparing the time management information added to the packet of the certain information with the offset-considered time management information converted by the conversion unit. And extracting means for extracting a packet as a simultaneous packet set, and connecting the packets constituting the simultaneous packet set, based on the time management information. And a multiplexing means for connecting each simultaneous packet set in time order. 39. A packet sequence formed by packetizing a code sequence of a certain information and a packet sequence formed by packetizing a code sequence of an arbitrary information having the same time property as the certain information, and recording the multiplexed data on a recording medium. An offset position for the certain information is specified with respect to the arbitrary information, the time management information added to the packet of the arbitrary information is taken into consideration by the offset considering the offset position. A conversion unit for converting the time management information into time management information; a packet having time management information at the same time by comparing the time management information added to the packet of the certain information with the offset-considered time management information converted by the conversion unit. Extracting means for extracting the same as a simultaneous packet set, and a time based on the time management information for the simultaneous packet set extracted by the extracting means. A code recording apparatus, comprising: an arithmetic unit for obtaining a target-relationship relationship; and a multiplexing unit for connecting packets constituting the simultaneous packet set and connecting the simultaneous packet sets in chronological order with reference to the relationship. 40. Claim 35, Claim 36, Claim 3
8. The code recording device according to claim 39, wherein the extraction unit performs the extraction by discarding a predetermined number of lower-order bits of the time management information and referring to data of a predetermined upper-order number of bits. 41. Claims 35, 36, and 3
8. The code recording apparatus according to claim 39, wherein the extraction unit rounds a predetermined number of lower bits of the time management information and refers to data of a predetermined upper bit to perform the extraction. 42. Claim 34, Claim 36, Claim 3
7. The time management information according to claim 39, wherein the time management information is decoding time information indicating a decoding time of an access unit and reproduction time information indicating a reproduction time. A code recording device, wherein a temporal pre / post relationship is obtained based on any of the reproduction time information. 43. The time management information according to claim 35, wherein the time management information is decoding time information indicating a decoding time of an access unit and reproduction time information indicating a reproduction time. A code recording device, wherein each simultaneous packet set is linked based on either the decoding time information or the reproduction time information. 44. The code recording apparatus according to claim 34, wherein the certain information includes at least moving image information, and the arbitrary information is audio information. 45. The code recording apparatus according to claim 42, wherein the certain information includes at least moving image information, and the arbitrary information is audio information. 46. The decoding method according to claim 42, wherein the decoding time information is a MPEG1 or MPEG2 standard.
The playback time information is a TS (Decoding Time Stamp), and the playback time information is a PTS (Presentati
on Time Stamp). 47. The code recording according to claim 34, wherein the certain information is moving image information, audio information, and subtitle information, and the arbitrary information is another audio information. apparatus. 48. The code recording apparatus according to claim 42, wherein the certain information is moving image information, audio information, and caption information, and the arbitrary information is another audio information. 49. The code recording apparatus according to claim 44, wherein a data amount of each access unit of the code sequence of the moving image information is allowed to be different for each access unit. . 50. The code recording apparatus according to claim 44, wherein the code sequence of the moving image information is a code sequence of a variable bit rate. 51. A code multiplexing method for multiplexing code strings of a plurality of types of information having the same time, wherein a code for an access unit of each code string is determined based on position information indicating a position in each code string. A code multiplexing method that obtains a temporal pre / post relationship that is not tied to a sequence, packetizes each of the code strings, and multiplexes each packet in time order based on the pre / post relationship. 52. A code multiplexing method for multiplexing a plurality of types of code streams of information having the same time, wherein a code is assigned to an access unit of each code stream based on position information indicating a position in each code stream. Obtain a temporal predecessor-relationship that is not bound by a column, and packetize each of the code strings. If the target packet includes the head of an access unit during packetization, the position information of the head access unit in the target packet is used. A code multiplexing method, comprising: generating time management information based on the information, adding the time management information as header information of the target packet, and connecting the respective packets in time order based on the preceding and following relationship. 53. A code multiplexing method for multiplexing a plurality of types of code strings of information having the same time, wherein each of the code strings is packetized, and a target packet includes a head of an access unit at the time of packetization. In this case, time management information is generated based on the position information of the first access unit in the target packet and added as header information of the target packet, and the generated packet is classified into the type based on the time management information. A code multiplexing method that obtains a temporally preceding and succeeding relationship that is not obscured, and connects each packet in time order with reference to the preceding and following relationship. 54. A code multiplexing method for multiplexing a plurality of types of code strings of information having the same time, wherein each of the code strings is packetized, and a target packet includes a head of an access unit at the time of packetization. In this case, time management information is generated based on the position information of the first access unit in the target packet and added as header information of the target packet, and the time management information of the generated packet is used between the different types of information. Extracting packets having the same time management information at the same time as a simultaneous packet set, obtaining a temporal predecessor-relationship based on the time management information with respect to the extracted simultaneous packet set, and forming the simultaneous packet set And multiplexing the simultaneous packet sets in chronological order with reference to the preceding and following relations. 55. A code multiplexing method for multiplexing code strings of a plurality of types of information having the same time, wherein an arbitrary information in the plurality of types of information is specified with an offset position with respect to other information, The position information indicating the position of the information in the code string is converted into offset-considered position information in consideration of the specified offset position, and the code of the offset-considered position information and the other information in the code string of the arbitrary information is converted. A time-preceding relationship that is not restricted to a code sequence is determined for an access unit of each code sequence based on position information indicating a position in the sequence, and each of the code sequences is packetized, and each packet is determined based on the preceding-rear relationship. A code multiplexing method that multiplexes in time order. 56. A code multiplexing method for multiplexing a plurality of types of information code streams having the same time, comprising: specifying an offset position with respect to other information with respect to arbitrary information in the plurality of types of information; The position information indicating the position of the information in the code string is converted into offset-considered position information in consideration of the specified offset position, and the code of the offset-considered position information and the other information in the code string of the arbitrary information is converted. Based on the position information indicating the position in the sequence, the temporal sequence of the access units of each code sequence is determined without regard to the code sequence, and each of the code sequences is packetized. In the case of including the head of the packet, the time management is performed based on the position information or the offset-considered position information of the head access unit in the target packet. Generates information added as the header information of the target packet, each packet is linked in chronological order based on the destination after relationship, code multiplexing method. 57. A code multiplexing method for multiplexing a code sequence of a plurality of types of information having the same time, wherein an arbitrary information in the plurality of types of information is designated with an offset position with respect to other information, The position information indicating the position of the information in the code string is converted into offset-considered position information in consideration of the specified offset position, and each of the code strings is packetized. If the packet includes the head, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet, and added as header information of the target packet. Based on the management information, a time-preceding relationship that is not restricted to the type is obtained, and the respective Tsu the coupling chronologically DOO, code multiplexing method. 58. A code multiplexing method for multiplexing a code string of a plurality of types of information having the same time characteristic, wherein an arbitrary information in the plurality of types of information is designated with an offset position with respect to other information, The position information indicating the position of the information in the code string is converted into offset-considered position information in consideration of the specified offset position, and each of the code strings is packetized. If the packet includes the head, time management information is generated based on the position information or offset-considered position information of the head access unit in the target packet and added as header information of the target packet, and the time management of the generated packet is performed. By comparing information between different types of information, packets having time management information at the same time are regarded as a simultaneous packet set. Extracting, obtaining a temporal predecessor-relationship based on the time management information with respect to the extracted simultaneous packet set, connecting the packets constituting the simultaneous packet set and referring to the precedent-rear relation, and chronologically reordering each simultaneous packet set Concatenation, code multiplexing method. 59. A code multiplexing method for multiplexing a code string of arbitrary information having the same time characteristic as a certain information in a packet string of certain information, wherein an offset position of the arbitrary information with respect to the certain information is provided. Converting the position information indicating the position of the arbitrary information in the code string into offset-considered position information in consideration of the specified offset position, and packetizing the code string of the arbitrary information, When the target packet includes the head of the access unit at the time of conversion, time management information is generated based on the offset-considered position information of the head access unit in the target packet, and added as header information of the target packet. Information based on the time management information added to the information packet and the time management information added to the arbitrary information packet. Seek time antecedent relationship packet agnostic like, connecting with reference to the destination after the relationship in order to each packet time, code multiplexing method. 60. A code multiplexing method for multiplexing a code string of arbitrary information having the same time characteristic as a certain information in a packet string of certain information, wherein an offset position of the arbitrary information with respect to the certain information is provided. Converting the position information indicating the position of the arbitrary information in the code string into offset-considered position information in consideration of the specified offset position, and packetizing the code string of the arbitrary information, When the target packet includes the head of the access unit at the time of conversion, time management information is generated based on the offset-considered position information of the head access unit in the target packet, and added as header information of the target packet. The time management information added to the information packet and the time management information added to the arbitrary information packet are different types of information. A packet having the same time management information at the same time is extracted as a simultaneous packet set, and the packets constituting the simultaneous packet set are connected together, and each simultaneous packet set is connected in time order based on the time management information. To do the code multiplexing method. 61. A code multiplexing method for multiplexing a code string of arbitrary information having the same time property as a certain information in a packet string of certain information, wherein an offset position of the arbitrary information with respect to the certain information is provided. Converting the position information indicating the position of the arbitrary information in the code string into offset-considered position information in consideration of the specified offset position, and packetizing the code string of the arbitrary information, When the target packet includes the head of the access unit at the time of conversion, time management information is generated based on the offset-considered position information of the head access unit in the target packet, and added as header information of the target packet. The time management information added to the information packet and the time management information added to the arbitrary information packet are different types of information. A packet having time management information at the same time is extracted as a simultaneous packet set by comparing between them, and a temporal predecessor relationship is obtained for the extracted simultaneous packet set based on the time management information, thereby forming the simultaneous packet set. A code multiplexing method, wherein packets are connected, and respective simultaneous packet sets are connected in time order with reference to the preceding and following relations. 62. A code multiplexing method for multiplexing a packet sequence of arbitrary information having the same time synchronization with a certain information packet sequence, the offset position of the arbitrary information item with respect to the certain information item. And converting the time management information added to the packet of the arbitrary information into offset-considered time management information in consideration of the specified offset position, and the time management information added to the packet of the certain information. A code multiplexing method comprising: obtaining a temporal pre / post relationship of packets irrespective of the type of information based on the converted offset-considered time management information; and connecting the packets in chronological order with reference to the pre / post relationship. 63. A code multiplexing method for multiplexing a packet sequence of arbitrary information having the same time as the certain information in a packet sequence of the certain information, wherein an offset position of the arbitrary information with respect to the certain information is provided. And converting the time management information added to the packet of the arbitrary information into offset-considered time management information in consideration of the specified offset position, and the time management information added to the packet of the certain information. And comparing the converted offset-considered time management information with a packet having the time management information at the same time as a simultaneous packet group, and connecting the packets constituting the simultaneous packet group, based on the time management information. A code multiplexing method in which each set of simultaneous packets is connected in time sequence. 64. A code multiplexing method for multiplexing a packet sequence of arbitrary information having the same time characteristic as a packet sequence of certain information, wherein an offset position of the arbitrary information with respect to the certain information is provided. And converting the time management information added to the packet of the arbitrary information into offset-considered time management information in consideration of the specified offset position, and the time management information added to the packet of the certain information. Comparing the converted offset-considered time management information with the converted offset-considered time management information to extract a packet having time management information at the same time as a simultaneous packet set, Connecting the packets constituting the simultaneous packet set, and connecting the simultaneous packet sets in time order with reference to the preceding and following relations. To, code multiplexing method. 65. Claim 54, Claim 58, Claim 6
0, the predetermined number of lower-order bits of the time management information is truncated, and the extraction is performed by referring to data of a predetermined upper-order number of bits. Multiplexing method. 66. Claim 54, Claim 58, Claim 6
0, the predetermined number of lower bits of the time management information are rounded, and the extraction is performed by referring to data of a predetermined upper number of bits. Multiplexing method.