JP3536493B2 - Authoring system, encoder and multiplexer used in the system, and method for generating multiple bit streams - Google Patents

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to, for example, MPEG2.
, An authoring system for generating a multiplex bitstream according to a digital moving image coding standard and recording it on a storage medium, and an encoding device, a multiplexing device, and a multiplex bitstream used in this system On how to do.

More specifically, for example, all kinds of video materials such as movie videos and music videos are targeted, and images of the video materials and background sounds (including surround, if desired) corresponding to the videos, and a plurality of background sounds. Related information such as dubbing of languages, subtitles in multiple languages, MPEG2 standard,
Individually encoded bitstreams generated in accordance with the MPEG audio standard, etc. are multiplexed and generated in a multiplex bitstream format, which is used when digitally recording on storage media such as optical discs, magneto-optical discs, and magnetic tapes. The present invention relates to an authoring system, a coding device and a multiplexing device used in this system, and a method for generating a multiple bit stream.

Such an authoring system is generally composed of an encoding device, a multiplexing device, etc., and the encoding device compresses and encodes source data such as video, background sound, and audio according to MPEG2. This compression coded data is generated by a multiplexing device into one multiplexed bit stream and recorded in a storage medium.

[0004]

2. Description of the Related Art [MPEG for multiplexing multimedia]
System] First, in order to facilitate understanding of the present invention, an outline of an MPEG2 (Moving Picture Experts Group 2) system for multiplexing multimedia related to the present invention and a finally generated multiple bit stream will be described. To do. Publications related to such multiple bit streams include: Hiroshi Yasuda, "International Standards for Multimedia Coding," Maruzen Co., Ltd. (1991), Research Group for Multimedia Communications: "Latest MPEG Textbooks," ASCII Publishing (1995). ), The Television Society of Japan: “Image Information Compression” by Ohmsha (1991), etc.
These are appropriately cited with respect to the description of the G system.

(MPEG system) MPEG video and M
A coded stream (bit string) of PEG audio and further another coded stream are integrated into a single stream including synchronization, and the stream is converted into a data format suitable for the unique format of the storage medium. There is a need. Further, it is required to add data used by the application. Thus, the role of the MPEG system is to synchronize and multiplex video and audio.

The MPEG standards include MPEG1 which is a standard mainly for storage media such as CD-ROM and MPE.
There is MPEG2, which is a standard for a wide range of applications, including G1 applications. The present invention mainly uses MPEG2. Also, MPEG2
Regarding system multiplexing, a program stream (program stream) that constitutes one program (program) is
am, MPEG2-PS) and a transport stream that can configure multiple programs at the same time
m, MPEG2-TS), but of these, the present invention mainly targets MPEG2-PS.

FIG. 6 is a system configuration diagram for understanding the basic functions of the MPEG2 system related to the present invention. This system uses a video encoder (encoder) 3
And an encoder 5 including an audio encoder (encoder) 5 and a multiplexer (MUX Multiplexer) 21 for multiplexing the outputs of the video encoder and the audio encoder.
And a storage medium 53 for recording the multiplex bitstream from this multiplexer and a demultiplexer (DMUX Demultiplexer) for separating the multiplex bitstream recorded on the storage medium.
multiplexer) 55, and a decoding device 60 including a video decoder 57 and an audio decoder (decoder) 59 for decoding the separated encoded data, respectively.

With regard to the operation of this system, the video encoder 3 and the audio encoder 5 of the encoding device 1 individually encode and compress individual video and audio materials in accordance with the MPEG standard, while maintaining their cooperation. To do. The individual encoded streams (data strings) are multiplexed by the multiplexing device 21 according to the application, multiplexed according to the format of the storage medium of the stream, and recorded. That is, a plurality of individual coded streams are time-division multiplexed into one stream.

In the storage medium 53 in which the multiple streams are recorded, the separating device 55 separates each individual stream (data string) portion such as video and audio, and sends it to each decoding device 60. The video decoder 57 of the decoding device 60,
The audio decoder 59 and the like synchronously decode the individual stream on the decoding device side, as intended on the multiplexing device side. Each of the decoders 57 and 59 individually decodes the encoded bit stream and outputs it to an output device (television monitor, speaker, etc.).

In recent years, a DVD (Digital) using an optical disk, a magneto-optical disk or the like as a recording medium as a transmission medium 53 for transmitting a multiple bit stream from the encoding / multiplexing devices 1, 21 to the demultiplexing / decoding devices 55, 59. Vi
deo Disc) is being considered. The present invention is an invention made in relation to this DVD.

(Multiplexing / Demultiplexing / Synchronizing Method) Next, the multiplexing / demultiplexing / synchronizing method will be schematically described. Packets used in many time division multiplexing systems (Packe
The multiplex method by t) is also adopted in the MPEG2 system. FIG. 7 shows an MPEG2-PS (program stream) reference decoder incorporating a basic packet-based one-program multiplexing / demultiplexing method. Details will be described later.

[0012] The packet multiplexing is, for example,
When multiplexing data such as video and audio, divide each video and audio into bitstreams (bit strings) of appropriate length called packets, attach additional information such as headers, and add video and audio as appropriate. This is a method of switching the data packet of and recording in time division. As shown in FIG. 8 (1), these packets include
At the head part called a header, there is information for identifying attributes such as video and audio. Also, if necessary, a CRC (Cyclic R) for detecting a bit error at the end is provided.
edundancy Check) is added.

The packet length strongly depends on the storage medium. For example, as long as the optical disk system (40
There is also a packet length of 96 bytes), and the upper limit is about 2 ¹⁶ (64 Kbytes) in MPEG, and each packet may be fixed length or variable length in order to have flexibility. Since the fixedly required part such as the header does not depend on the packet length, the short packet has a large overhead (additional data for multiplexing) and the recording efficiency decreases, but the switching time of time division multiplexing is short. In addition, there is an advantage that the delay due to multiplexing and the buffer memory amount can be small.

As shown in FIG. 8 (2), MPEG2-P
In S (program stream), there is a layer called a pack layer above the packet of video, audio and other data. Normally, a plurality of packets are handled in a unit called a pack. FIG. 8B shows the structure of the pack. In the pack header portion, there is additional information for reference to a time reference for synchronous reproduction, which will be described later. The main purpose of the pack is to enable decoding and reproduction from the middle of the stream.

(AV synchronization system by time stamp) In the decoding device, the individual stream is decoded in units of access units. For example, in the video 1
For frame and audio, one audio frame
Become an access unit. With regard to the MPEG2 synchronization method, information called a time stamp indicating when to decode and reproduce is added to each access unit of video, audio, and other data. For this time stamp, a time reference is given by information called SCR (System Clock Reference).

The time stamp is a tag for managing the time of decoding / reproduction processing that is attached to each access unit.
Equivalent to. There are two types of time stamps due to the MPEG audio encoding method, one is PTS.
(Presentation Time Stamp) is the "time information of reproduction output management", and the other is DTS (Decoding Time Stamp).
This is "decoding time management information" called p). These time stamps are added to the packet header if the access unit is at the beginning of the packet. However, if there is no access unit at the beginning of the packet, no time stamp is added to the packet header. Also, even if there are two or more access units at the head of the packet, only the time stamp corresponding to the first access unit is added to the packet header.

The MPEG2-PS (program stream) reference decoder shown in FIG. 7 will be described. The multiplex bitstream recorded on the storage medium is separated into individual streams by the separating device 55. The video bitstream is a video STD (System Target Deco
der, the decoder corresponding to the system) buffer 57 ', and the audio bit stream is stored in the audio STD buffer 59'. The SCR (system time reference reference value) that gives a time reference to the time stamps (PTS, DTS) is stored in the system decoder 61.

PTS (reproduction output time management information) is S in the MPEG2-PS reference decoder shown in FIG.
When TC (System Time Clock, a basic synchronization signal, which basically matches SCR) matches PTS, the access unit is reproduced and output. PT
The accuracy of S is 33 when the value measured with the clock of 90 kHz.
Expressed in bit length. The reason for 90 kHz is NTSC, P
This is because it is a common multiple of the frame frequency of both AL video systems, and is more accurate than one sample period of audio. The reason for the 32-bit length is 90 kHz
z SMPTE (Soci
ety of Motion Picture and Television Engineers,
This is because it is possible to express the range of 24 hours a day like the time code of the American Film and Television Engineering Association.

DTS (decoding time management information) is MPE
G2 is provided because there is a special video encoded stream transmission order. That is, I picture (Intr
a-Picture, intra-frame coded image and P picture (Pr)
edictive-Picture, inter-frame forward prediction coded image)
Is a B-picture (Bidirectinally Predictive-Picture
, Bidirectional predictive coded image), the order of decoding is different from the order of reproduction and output in order to send it to the coded stream in advance. PTS and DT
If S is different, add both time stamps,
If they match, only the PTS is to be added. Specifically, in a video coded stream having B pictures, PTS and DTS are used for I pictures and P pictures.
For B pictures, I pictures without B pictures and B pictures, only PTS is required.

SCR (System Clock Reference system time reference value) is intended for the encoder side with respect to the value of STC (reference synchronization information) which is the time reference in an MPEG system decoding device including video and audio decoders. This is information for setting / calibrating to the specified value. Also, SC
When using R, the SCR value alone is not sufficient, and the accuracy of the timing of the bytes (arrival time at the decoder) in the stream carrying the SCR value is required.
The SCR is sent in 6 bytes (actual data is 42 bits) in MPEG2, but on the decoder side, the STC is required to set the value of the SCR at the moment of reaching the final stage.

In response to this SCR, the decoder side STC
It is possible to construct a PLL (Phase Locked Loop) integrated with the STC and to have the STC whose frequency is completely matched with the system clock of the encoder in the decoder.

The separating device 55 and the decoding device 60 will be further described. The separating device 55 separates the multiplexed bit stream into individual video streams, individual audio streams, and the like.
The data is sequentially recorded in the respective STD buffers 57 'and 59'. Each decoder reads a bit stream of one access unit (one frame for video and one audio frame for audio) from a corresponding STD buffer at a predetermined timing and performs a decoding process.

FIG. 9 is a diagram for explaining the state of being stored in the STD buffer and the timing of reading and decoding at a predetermined timing. Initially, assume that the STD buffer was empty. Individual bitstreams are sequentially accumulated from the separation device. When one frame has been accumulated, the decoding device reads the accumulated bitstream by one access unit at a time, and starts the decoding process of the read bitstream. If the read timing is delayed from the predetermined timing or the bit rate is too fast, the STD buffer overflows. If the timing is too early or the bit rate is too slow, the underflow will occur and the buffer will fail in any case.

Further, in the multiple bit stream, it is necessary to synchronize the corresponding audio and the like with the video in an appropriate order and ratio. For example, if only the video is sent and the corresponding audio or the like is not sent, the video and the audio cannot be synchronized and cannot be reproduced.

Therefore, when the video data and the audio data are multiplexed in advance by the multiplexer, the ST
It is necessary to generate a multiple bit stream while performing scheduling so that the D buffer does not break down and that video and audio are synchronized and decoded.

Conventionally, a real-time authoring system used for a video CD as shown in FIG. 10 is known. The real-time authoring system of FIG. 10 comprises a coding device 1 and a multiplexing device 21. The coding device 1 generally comprises a video encoder 3 for coding a digital video source signal (individual material) and a digital encoder. An audio encoder 5 that encodes an audio source signal (individual material).

The multiplexer 21 has a buffer memory 65 connected to the video encoder 3 and a buffer memory 67 connected to the audio encoder 5 in the front stage, and these buffer memories in the rear stage. And a multiplexer circuit 69 connected to 65 and 67.

In the operation of the real-time authoring system shown in FIG. 10, the video encoder 3 and the audio encoder 5 individually compress individual materials such as a video source signal and an audio source signal while maintaining their cooperation. Encode. The coded video bitstream is sequentially stored in the buffer memory 65, and similarly, the coded audio stream is sequentially stored in the buffer memory 67.

The multiplexing circuit 69 multiplexes the video data stored in the buffer memory 65 and the audio data stored in the buffer memory 67 in a predetermined order so that the video bit stream and the audio bit stream are time-divided. Generate a multiplexed multiple bitstream. The multiple bitstream is recorded on a suitable storage medium.

[0030]

When a conventional multi-bitstream is generated using such a conventional real-time authoring system, there are three major problems. That is, there are problems regarding multiplexing scheduling, problems when multiple times multiplexing is performed using the same elementary stream, and problems regarding reference image position information.

(1) Problems with Multiplexing Scheduling In the past, only video and audio were multiplexed as individual materials, but as the scope of application expands, all video sources including movie sources, music videos, etc. are targeted. There is. For example, corresponding to one video, audio is added with surround sound and accumulated, or dubbed in several languages, and a multiple bit stream including subtitles in several languages is generated to store the media. Occasionally you want to record.

In this case, an individual bit stream obtained by compressing a moving image signal, an audio signal, a caption signal, etc. by a respective predetermined encoding device, and a non-compressed bit stream etc. are multiplexed (MUX). Multiplex with 1 MP
It is necessary to record it on a predetermined recording medium as an EG2 standard multiple bit stream.

The MPEG2 standard multi-bitstream is a demultiplexer (DMU) in the preceding stage of the demultiplexer / decoder.
X) separates into multiple types of individual bitstreams,
Each individual bit stream is decoded and reproduced by each decoding device in the subsequent stage. The individual bitstreams separated / sent from the separation device are accumulated in the respective STD buffers, and the bitstreams for access units are read and decoded by the respective decoding devices.

In this case, each elementary stream separated / sent from the separation device should not overflow or underflow its STD buffer and be destroyed. In addition, it is necessary that audio and the like are combined with the video in a synchronized ratio. Therefore, in order to prevent the STD buffer of each decoding device from failing, a scheduling process such as appropriate combination of individual bit streams and determination of an appropriate individual bit stream length is performed in advance when multiplexing individual bit streams in a multiplexing device. Must be done.

In order to determine the size of the portion to be multiplexed (packetized) at the time of performing the scheduling process of multiplexing, the elementary stream (prime bit string) to be multiplexed is prefetched and By observing the size of the access unit in the previous part (the part with the slower DTS), it is possible to determine the current multiplexing.

Therefore, when the encoding and multiplexing are executed in real time as in the conventional real-time authoring system, it is necessary to equip the multiplexer with a buffer memory for pre-reading of the multiplexing scheduling.

With respect to this buffer memory, in the authoring system, a large number of elementary streams may be multiplexed as the scope of application is expanded. In actuality, assuming such a case, MPEG2 permits multiplexing of a maximum of 32 elementary streams. In order to process the multiplexing of such a large number of elementary streams in real time, as a matter of course, a huge capacity of buffer memory is required.

Further, as the number of elementary streams to be multiplexed increases, the pre-reading is performed in order to perform the multiplexing scheduling so that the STD buffer of each decoding device will not be broken as described above. The range of will have to be expanded considerably. From this point of view, the multiplexer requires a huge capacity of buffer memory.

Further, as the number of elementary streams increases, the amount of analysis also increases, and the load on the multiplexer increases.

(2) Problem of Repeating Multiplexing Multiple Times Using Elementary Streams of the Same Video According to the MPEG2 multiplexing specification, up to 32 audio elementary streams can be multiplexed. Using this audio multiplexing tolerance range, a movie or video source can be dubbed in one language in multiple languages, or subtitles in multiple languages can be inserted to generate multiple bitstreams. , It is desirable to create a storage medium that stores this.

In other words, one program (program)
This is a case where it is desired to create a plurality of multiplex bit streams in which the combination is changed from the plurality of audio elementary streams and the plurality of subtitle elementary streams for the video elementary stream of.

For example, with respect to a certain movie, software in which English voice and Japanese voice are multiplexed and also with English subtitle and Japanese subtitle, and English voice and Spanish voice and English subtitle and Spanish subtitle are used. You may want to create two types of software, including multiplexed software.

In such a case, in order to create different kinds of multiple bit streams for one program,
Encoding the same video, audio, and subtitles separately for each generation of each multiplex bitstream and analyzing these elementary streams for multiplex scheduling causes duplication of work and Is useless and very inefficient.

(3) Problems concerning reference image position information In the MPEG2 multiple stream, PSD (program stream directory) and PSM (program stream map) which are PSI (program specification information) described later are provided in pack units. A special packet called
Inserted after pack header. Offset values to I-pictures to be referenced before and after the pack are stored in PSD. The PSM mainly describes multiplexed elementary streams and their mutual relationships, but other information can be freely incorporated, such as music videos and movies. It is possible to enter an offset value or the like to the I picture at the beginning of the track in the same position.

In PSD, if the range of I-pictures to be referred to in the future is expanded to be referred to in the past, the contents of the PSD are determined and multiplexed unless the pre-reading range of the elementary stream is expanded accordingly. You cannot write to the bitstream.

Since a track is usually a time unit of several minutes to several tens of minutes, in reality, while multiplexing, a huge amount of data is prefetched to determine information such as the address of the I picture at the head of the track. , PSM data cannot be written.

An object of the present invention is to provide an authoring system which solves the above problems and efficiently multiplexes a large number of individual data.

The present invention further provides an encoding means and a multiplexing means used in such an authoring system.

[0049]

SUMMARY OF THE INVENTION An authoring system according to the present invention comprises an elementary stream obtained by encoding a plurality of individual source signals in accordance with, for example, MPEG2 or MPEG audio, and other digital signals. An authoring system that multiplexes columns and generates a multiplex bitstream, and an encoding device that simultaneously generates each elementary stream as a file at the time of the encoding, and at the same time generates the associated auxiliary information as an intermediate file, respectively. A multiplexing device is provided which performs multiplexing scheduling from the files of each elementary stream and the intermediate file without analyzing the elementary streams to generate a multiplexed bitstream.

Further, in the authoring system according to the present invention, in the above-mentioned authoring system, the multiplexing device has a scheduling unit, a reference image position information generating unit, and a multiple bit stream generating unit, and the scheduling unit is , Scheduling each elementary stream to be multiplexed, and generating the result as an intermediate file of schedule data, the reference image position information generator, all the multiple bitstreams constituting the intermediate file of the schedule data. An intermediate file of reference image position information to be inserted for each pack is generated, and the multiplex bitstream generator multiplexes from each elementary stream based on the information of the intermediate file of the schedule data and the intermediate file of the reference image position information. bit It is generating the stream.

Further, in the authoring system according to the present invention, in the above-mentioned authoring system, the size of each access unit, which is at least the minimum unit of multiplexing, DTS and P in the intermediate file of the coded accompanying information.
TS is described.

Further, the method for generating a multiple bit stream according to the present invention is characterized in that each elementary stream obtained by encoding a plurality of individual source signals including a video source signal, an audio source signal, and a caption source signal, A method of generating a multiplex bit stream by multiplexing with other digital signal sequences, wherein, in the encoding step, each elementary stream file is generated and at the same time each encoded additional information is used as an intermediate file. In the step of generating and multiplexing, the intermediate file is fetched to perform multiplexing scheduling without performing analysis of the elementary stream, thereby performing multiplexing.

Further, the method for generating a multiple bit stream according to the present invention is the above-described method for generating a multiple bit stream, wherein the multiplexing step includes a scheduling step, a reference image position information generating step, and a multiple bit step. And a step of generating a stream, wherein the step of scheduling schedules each elementary stream based on the intermediate file of the encoding accompanying information and generates the result as an intermediate file of schedule data, and the reference image position information. In the step of generating an intermediate file of reference image position information to be inserted for every pack forming a multiplex bitstream from the intermediate file of the schedule data, and in the step of generating the multiplex bitstream, The middle fa Generating multiple bit streams from each elementary stream based on the Le and information intermediate file of the reference image position information.

According to the encoding / multiplexing apparatus and the method for generating a multiple bit stream according to the present invention as described above, an encoding processing step, a scheduling processing step,
It is separated into multiplex processing steps, and multiplex processing can be performed without analyzing each elementary stream.

In addition, since each elementary stream file and each encoded incidental information file are generated in the encoding processing of each individual source signal, it is possible to generate an arbitrary combination of multiple bit streams later.

[0056]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the accompanying drawings. It should be noted that the same elements will be denoted by the same reference symbols and redundant description will be omitted.

[Data Structure and Bitstream Stream Structure of MPEG System] First, the data structure of a multiple bitstream generated by the authoring system of the present embodiment will be described, and then PSI (Program Specific I).
nformation, program specification information).

The data structure of the multiple bit stream of this embodiment conforms to MPEG2-PS (Program Stream), and the data structure and bit stream have a hierarchical structure as follows. Although this MPEG stream has a large number of 1-bit flags and the like, it is a byte stream that is byte-aligned for each unit such as a header, and information indicating the length precedes the data portion that is not fixed length. If you don't need it, you can skip that part.

(MPEG2-PS Data Structure) The data structure of the multiple bit stream (MPEG2-PS) has the upper layer (pack layer) and the lower layer (packet layer).
The layers are shown separately in FIG. 1 and FIG. Regarding the upper layer (pack layer) shown in FIG. 1, one program may be referred to as a sequence, but it starts at the top "pack header" and ends at "end code". As described above, the pack is generally composed of a plurality of packets, and the pack at the head of the program has an information group called "system header" that describes the outline of the entire stream (parameter information). This system header is required to be added to the first pack, but it is an option in the second and subsequent packs.

The pack header has a variable length, which is a 32-bit (4 byte) "pack start code" followed by 2 bits "01", which is an identification code with MPEG1. After that, the above-mentioned "SC
R (system time reference value) "and information indicating the" program multiplexing rate "(incremental value in the case of variable bit rate) of this stream in units of 50 bytes / second.

Since up to 7 bytes of stuffing bytes can be placed, the length is indicated by "pack stuffing byte length" prior to this stuffing byte. Following this, there are M bytes (maximum 7) of "pack stuffing bytes" (the stuffing bytes will be explained later). From the above, the pack header length is variable from 14 bytes to 21 bytes.

The system header contains a 32-bit "system header start code" followed by this system header.
“Header length” is included as an information item (field) indicating the length of the header itself. In the actual data portion, "bit rate", "number of audio channels", four types of "flags", "number of video channels" (even one program can have multiple videos and audios). And the buffer required for each individual trim.
The fields "P-STD buffer area scale" and "P-STD buffer area size" indicating the memory amount are included. The flag has the following four items. “Fixed rate flag”: Identification of a constant bit rate. "CSPS Flag (Constrained System Parameter Str
eam, restrictive system parameter stream) "
… By limiting the parameters of the system stream,
It is intended to achieve compatibility with actual decoders. "Audio lock flag" ... The audio in the system stream is the system clock (90 kHz)
And a flag indicating whether or not it is phase locked. "Video lock flag" ... A flag indicating whether the video in the system stream is phase-locked with the system clock (90 kHz).

Next, the "stream ID" shown in the "N times loop" in the system header is the last 8-bit portion of the 32-bit packet start code, as shown in the table below. Streams are distinguished by these 8 bits, and a parameter (required buffer memory capacity) is indicated for each individual stream.

Stream ID Table (Part) Stream ID Stream Type 1011 1100 Program Stream Map (PSM) 1011 1101 Private Stream 1 1011 1110 Padding Stream 1011 1111 Private Stream 2 110X XXXX MPEG Audio Stream 1110 XXXX MPEG Video stream 1111 0010 DSM ^* control command 1111 1111 Program stream directory (PSD) (*) DSM: Digital Storage Media

Lower Layer (Packet Layer) MPEG packets are PES packets (Peckeized Elem).
entary Stream Packet) and this structure is MPEG
Since it is used in common with 2-TS, the header part is complicated, but it is easy to understand hierarchically. Flags are put together in the first part, followed by optionally condition-coded items corresponding to those flags. Further, the option item has a secondary option item.

The 32-bit "packet start code" is
It consists of a 24-bit fixed part "packet start prefix" and an 8-bit "stream ID (identification)" part. In MPEG2-PS, the stream ID is defined so that up to 16 channels of video and 32 channels of audio can be used.

The "packet length" indicates the data length of the packet following this field.

The system control information includes the following. 2-bit control code "10" following the packet length
Is used for distinguishing from "00", "01" or "11" of MPEG1 as described above. The “scramble control flag” can be used for confidential information, a commercial system, etc. by scrambling the data (recombining the data string, etc.). Usually, it is set to "00". "PES
"Priority flags" are important packets and those that are not (packets that can be discarded in some cases)
So that they can be distinguished. Usually, it is set to "0". Next, "data alignment" follows, and "copy / write"
Is information indicating the copyright of the encoded stream of video or audio. “Original / Copy” indicates whether it is an original or a copy. The “PES header length” indicates the length of the header of the PES packet.

The "option field" includes the following. "PTS" ... PTS (Presentation Time Stam)
This is "playback output time management information" called p). "DTS" ... DTS (Decoding Time Stamp) described above
It is "decryption time management information". Others such as “ESCR” and “ES rate” follow, but they are omitted because they are not directly related to this embodiment.

"Stuffing byte" and "Padding byte" of up to 7 bytes are dummy bytes and have no meaning in the sign. As an application, for example, in order to make the packet data length constant regardless of the presence or absence of PTS or DTS, a stuffing byte is placed instead of these in a packet without PTS or DTS.

In MPEG2-PS (with the data structure of PSI (program specification information)), a plurality of individual streams of video, audio, and other data are multiplexed into one, so that in a demultiplexing / decoding device. Information such as which packet to take out and how to decode it is necessary. These program specification information are collectively referred to as PSI (Program Stream Information). MPEG2-PS has two P
A packet for SI (program specification information) is defined. One is a PSM (Program Stream Map), and the other is a PSD (Program Stream Directory) packet. Data structures such as these are shown in FIGS. 11 to 15 and the difference from the above-mentioned PES packet will be explained.

A PSM (program stream map) packet is a packet describing additional information of a program and additional information of each individual stream. For example, information about video scalability encoding, audio language identification information, and the like can be described.

A PSD (program stream directory) packet is directory information for random access used in a storage medium, and is an entry point (for example, I picture for video) for random access for each individual stream. The leading address of the stream) is represented by the number of bytes from the beginning of the stream. The data structure of this packet is the normal P for the first half.
It is the same as the ES packet, but the latter half portion (the portion corresponding to the option data and the packet data) is different.

[Embodiment 1] In the case of creating a software in which a movie or video intended by the authoring system is made up of audio in several languages and subtitles in several languages, that is, a large number of elementary When a stream is encoded and these are time-division-multiplexed, the processing from encoding to multiplexing is performed in real time by performing multiplexing scheduling and reference image position information (PSM information, PSM information, P
This is unrealistic due to the problem of the read-ahead buffer memory capacity required for the generation of SD information).

When a plurality of types of software are created by changing the combination of video, audio, and subtitles to be multiplexed, the same encoding is performed each time, and the elementary stream is analyzed by the multiplexer. Is redundant and inefficient.

Therefore, in consideration of these problems, the authoring system of the present embodiment completely separates the encoding process and the multiplexing process in terms of time.

FIG. 4 is a diagram showing the system configuration of the authoring system according to the first embodiment. The authoring system generates the above-mentioned multiplex bitstream and stores it in the storage medium. This authoring system basically includes an encoding device 1 and a multiplexing device 21. However, in comparison with the configuration of the real-time authoring system described in connection with the related art, the configuration of the first embodiment is not the real-time method,
The encoding process, the process of generating information required for scheduling, and the multiplexing process are completely separated in time.

That is, a file of each elementary stream (a raw bit string) generated by encoding video, audio, and caption data with each encoding device, and an intermediate file of encoding additional information obtained in the encoding process. Is generated, and then an intermediate file of the schedule data generated by the scheduling device of the multiplexer is generated from this encoding accompanying information file, and further generated by the reference image position information generator from the intermediate file of this schedule data. An intermediate file of reference image position information is generated in advance, and then a multiplex bitstream generator is used to generate multiplex bitstreams using these intermediate files.

(Encoder) The encoder 1 has, for example, a video encoder 3, an audio encoder 5, and a caption data encoder 7.

A digital format video source signal output from a video camera or the like is input to the video encoder 3, and the video encoder 3 uses this video source signal as a moving image encoding standard such as the MPEG2 system. Is used for compression encoding. This video compressed data is accumulated in a storage medium such as a hard disk as an elementary stream (a raw bit string), and a video elementary stream file 9 is generated.

At the same time, the encoded incidental information obtained in the encoding step is accumulated in a recording medium such as a hard disk to generate the encoded incidental information file 11. This coded accompanying information file has not been used in the conventional authoring system, and is a new file created for the first time in this embodiment. The encoding incidental information is information required by a demultiplexing device that demultiplexes a multiplexed bitstream into individual stream parts such as video, audio, and caption data by the demultiplexing / decoding device in the subsequent stage described in the related art. Of the access unit, which is the minimum unit of DTS, DTS (decoding time management information), and PTS (reproduction output time management information).

Similarly, an audio source signal in digital format is input to the audio encoding device 5, and the audio encoding device 5 uses this audio source signal by using an audio encoding system such as the MPEG audio system. Compress and encode. This audio compressed data is an elementary stream (raw bit string)
Is stored in a storage medium such as a hard disk,
Audio Elementary Stream File 1
3 is generated.

At the same time, the encoded incidental information obtained in the encoding step is accumulated in a recording medium such as a hard disk,
The encoded accompanying information file 15 is generated. The encoding accompanying information is information required by a separating device that separates a multiple bitstream into individual bitstreams such as video, audio, and caption data in a separating / decoding device in a subsequent stage,
It consists of the access unit size and PTS.

Similarly, a subtitle source signal in digital format is input to the subtitle encoding device 7, and the subtitle encoding device 7 encodes this subtitle source signal. This subtitle coded data is accumulated as an elementary stream (a raw bit string) in a storage medium such as a hard disk, and a subtitle elementary stream file 17 is generated.

At the same time, the encoded incidental information obtained in the encoding step is accumulated in a recording medium such as a hard disk,
The encoded accompanying information file 19 is generated. The encoding accompanying information is information required by a separating device that separates a multiple bitstream into individual bitstreams such as video, audio, and caption data in a separating / decoding unit in a subsequent stage, and includes the access unit size, PTS, It consists of the display end time.

Note that FIG. 4 is an example, and the number of audio source signals and subtitle source signals is not limited to one each, and dubbing source signals in a plurality of languages and subtitle source signals in a plurality of languages are added. However, other source signals are possible. For these source signals, an elementary stream and an encoded ancillary information file are generated by the respective encoders. In order to simplify the explanation, the following video, audio,
I will continue to explain the three cases of subtitles.

It should be noted here that the conventional authoring system does not generate the encoded accompanying information file as described above, but prefetches and analyzes each elementary stream to analyze the information necessary for multiplexing. That is, the video, audio, and subtitle data were multiplexed together. However, in the case where the authoring system is intended to make a movie or video with audio in several languages and subtitles in several languages to create multiplexed software, that is, to encode a large number of elementary streams. In the case of multiplexing and multiplexing, it is unrealistic to perform the processing from encoding to multiplexing in real time due to the problem of the read-ahead buffer memory required for scheduling of multiplexing and generation of reference image position information. .

Therefore, in this embodiment, the real-time multiplexing work is not performed, and the video, audio, and subtitle data of one program (program) are encoded to the end to generate each elementary stream, and at the same time, this process is performed. Each of the coded incidental information collected in step 1 is generated as a coded incidental information file. By performing such processing, later, multiplexing processing of an arbitrary combination of a plurality of audio information, subtitle information, etc., for one video information is performed for each already-generated elementary stream and encoding. It becomes possible by using the accompanying information file.

(Multiplexing Device) The multiplexing device 21 includes a scheduling unit 23, a reference image position information generating unit 29,
It has a multi-bitstream generator 37.

(Scheduler) The scheduler 23 reads the coded incidental information files 11, 15, 19 and performs a scheduling process, and generates the result as an intermediate file of the schedule data 28.
The schedule data 28 is a pack information file 25 mainly relating to packs and packet information 27 mainly relating to packets.

The coded incidental information files 11, 15, and 19 read by the scheduling unit 23 store the size of the access unit, PTS, which is information necessary for scheduling, and in particular, DTS for video. Therefore, the scheduling unit 23 can analyze the lengths of packs of video, audio, and subtitle data without analyzing each elementary stream 9, 13, and 17.
Scheduled processing such as combination order becomes possible.

Through the scheduling process of the scheduling unit 23, these encoded accompanying information files 11,
Schedule data 28 is generated from 15, 19.
The schedule data 28 includes a pack information file 25 and a packet information file 27. Of these, the pack information file 25 is "SC
Information such as "R (system time reference value)", "program multiplexing rate", "stuffing bit length", and "packet number N in pack" is described. Pack header of the above-mentioned pack layer data structure, system
When these are determined among the information items of the header, other information items are basically uniquely determined by the system specifications.

These encoded accompanying information files 11 and 1
The packet information file 27 generated from Nos. 5 and 19 includes a “stream ID” (representing the type of elementary stream) and a buffer number (a plurality of streams having the same stream type) for each packet. (The number given to each one at one time)
Consists of. ), “Packet length”, “PES header length”,
Information such as “PTS (time management information for reproduction output)” and “DTS (time management information for decoding)” is described. Further, regarding the subtitles, information of "display end time" is described together with "PTS". Similarly, the PES packet
When the information items of the packet header of the layer data structure are determined, other information items are basically determined by the system specifications.

(Reference Image Position Information Generator) The reference position information generator 29 is a pack information file 25 which is schedule data 28 output from the scheduling unit 23.
Then, the packet information file 27 is read and the reference image position information 32 is generated. The reference image position information 32 is information to be inserted for every pack that forms a multiplex bit stream, and is the above-mentioned PSI (program specification information),
The reference image position information file is also one of the intermediate files. This reference image position information file is a PSD (program stream directory) information file 33.
And a PSM (program stream map) information file 35. The PSD information file 33 is information regarding the above-described PSD packet generation. Similarly,
The PSM information file 33 is information regarding the above-described PSM packet generation. This basically determines the PSD packet and the PSM packet.

(Multiple Bitstream Generator) The multiple bitstream generator 37 includes the files 9, 13 of the elementary streams generated by the encoders 3, 5, 7 for video, audio, subtitle data, etc. 17,
Pack information file 25, which is an intermediate file of the schedule data generated by the multiplexing scheduler 23
And the packet information file 27 and the PSD information file 33 and the PSM information file 35, which are intermediate files generated by the reference image position information generator 29, are combined to generate one multiplexed bit stream.

In the PSM packet, a description giving a description of the elementary stream and various information other than the start address of the assumed track and other information can be described. Please note that it is necessary to input to the generator 37.

The operation of the first embodiment will be summarized as follows. In the encoding process, the encoding is performed to generate the elementary streams as the files 9, 13 and 17, and the access required for the scheduling of the subsequent multiplexing is performed.・
Unit size, DTS (decoding time stamp, time management information for decoding), and PTS (pres for video)
The information such as the entation time stamp and the raw output time management information) is generated as the intermediate files 11, 15, and 19.

After the encoding process step is completed, the scheduling unit 23 of the multiplexer 21 performs the scheduling process from the intermediate files 11, 15 and 19 of the access unit information of each elementary stream of video, voice and character. The intermediate files 25 and 27 of the schedule data 28 of the packs and packets that form the multiplexed bitstream are generated.

Intermediate file 2 of schedule data 28
5 to 27, the reference image position information generator 29 outputs PSDs, Ps in all packs of the multiplex bitstream.
The reference image position information described in the SM is stored in the intermediate file 3
Generate as 3,35.

The multiple bit stream generator 37, based on the information in these intermediate files 25, 27, 33, 35, packs, PES packets, PSD packets, PSM.
While describing information in the packet, each elementary stream is combined in a predetermined order to perform time division multiplexing processing.

According to the first embodiment, not only the elementary stream itself but also all the access streams
By separately generating the unit information as an encoded accompanying information file, the scheduling process can be performed from the intermediate file of access unit information without analyzing each elementary stream on the multiplexing device side.

Therefore, even when a large number of elementary streams are multiplexed, the access unit information from the beginning to the end of each elementary stream is stored in each encoded accompanying information file. , The scheduling process can be performed only with the encoded accompanying information file, and the scheduling becomes very easy.

[Embodiment 2] Next, the elementary streams encoded by the encoding device and the associated encoded information describing the access unit size, DTS and PTS are managed as respective files. Thus, it is possible to generate a multiple bit stream by performing only the initial encoding and using the files without repeating the encoding thereafter and performing the multiplexing as many times as desired.

The system shown in FIG. 3 shows a case where Japanese audio and English audio are available as audio source signals for one video material. First, create a multiple bitstream of video, Japanese audio, and English audio.
After that, a case where a multiplex bitstream of only video and Japanese audio or a multiplex bitstream of only video and English audio is created is shown as needed.

FIG. 8 is a diagram showing a system configuration diagram of an authoring system according to the second embodiment shown in FIG. 3. This authoring system includes a coding device 1'and two voice multiplexing devices 21-1 and 21-2.

The coding device 1'includes a video coding device 3 and
It has speech coding devices 41 and 43.

A video source signal in digital format is input to the video encoding device 3, and the video encoding device 3
Compresses and encodes this video source signal using, for example, the MPEG2 system. This video compressed data is accumulated as an elementary stream in a storage medium such as a hard disk to generate a video elementary stream file 9.

At the same time, the coded incidental information obtained in the coding step is accumulated in a recording medium such as a hard disk,
The encoded accompanying information file 11 is generated. The coding incidental information is information required by the separating device that separates individual stream parts from the multiplex bit stream in the separating device in the subsequent stage, and includes the size of the access unit, the DTS, and particularly the PTS in the video.

A digital Japanese voice source signal is input to the voice encoding device 41, and the voice encoding device 41 compresses and encodes this Japanese voice source signal using an MPEG audio system. This Japanese audio compressed data is an elementary stream (raw bit string)
Is stored in a storage medium such as a hard disk,
Audio Elementary Stream File 4
5 is generated.

At the same time, the encoded incidental information obtained in the encoding step is accumulated in a recording medium such as a hard disk,
The encoded accompanying information file 47 is generated.

The English speech source signal in digital format is input to the speech coder 43, and the speech coder 4
3 compresses and encodes this English audio source signal using the MPEG audio system. This English audio compressed data is stored in a storage medium such as a hard disk as an elementary stream (elementary bit string), and a subtitle / elementary stream file 49 is generated.

At the same time, the encoded incidental information obtained in the encoding step is accumulated in a recording medium such as a hard disk,
The encoded accompanying information file 51 is generated.

The above is an example, and in addition to Japanese voice and English voice, dubbing in other languages, and subtitle source signals in other languages can be prepared. In this case, an encoder is prepared for each individual source signal, and a file of each elementary stream and encoding accompanying information is generated. To simplify the description, the video, Japanese voice, and English voice shown in the figure will be described below.

The multiplexers 21-1 and 21-2 are substantially the same as the multiplexer 21 of the first embodiment. Multiplexer 21
-1 multiplexes all of this information to generate a multiplexed bitstream of video, Japanese audio, and English audio. That is,
The multiplexer 21-1 has a scheduling device, a reference image position information generator, and a multiple bitstream generator,
The video elementary stream, the Japanese elementary stream, and the English elementary stream are multiplexed using the schedule data and the reference image position information.

Apart from this, it is also possible to generate a multiple bit stream of only video and Japanese audio. In this case, the already-encoded video and the Japanese audio file may be used for multiplexing by the multiplexer 23-2. This multiplexer 23-2 is also substantially the same as the multiplexer described in the first embodiment, that is, the multiplexer 21.
-1 by selectively changing the input
It can be 3-2.

Therefore, the multiplexer 23-2 also has a scheduler, a reference image position information generator and a multiple bit stream generator, and uses the schedule data and the reference image position information generator to generate a video elementary stream. Also, the Japanese elementary streams are multiplexed.

By managing the elementary stream and access unit information as an intermediate file in this way, for example, a plurality of audio for one movie material or a video material, dubbing in a plurality of languages, a plurality of languages By previously encoding subtitles and the like and generating each coded accompanying information file together with each elementary stream file, it is possible to later generate a multiple bit stream by an arbitrary combination. Duplication of the encoding process can be avoided, and the whole becomes very efficient.

[0118]

According to the present invention, it is possible to provide an authoring system for efficiently multiplexing a large number of individual data.

Further, according to the present invention, it is possible to provide an encoding means and a multiplexing means used in such an authoring system.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a data structure of a pack (upper) layer of MPEG2-PS to which this embodiment applies.

FIG. 2 is a P of MPEG2-PS to which this embodiment applies mutatis mutandis.
It is a figure explaining the data structure of an ES packet (lower) layer.

FIG. 3 is a diagram illustrating a data structure of an MPEG2-PS PSM packet and a PSD packet used as a reference image position information file in the present embodiment.

FIG. 4 is a diagram showing a configuration of an authoring system according to the first embodiment.

FIG. 5 is a diagram showing a configuration of an authoring system according to a second embodiment.

FIG. 6 is a diagram illustrating the configuration of a basic system of an MPEG system related to this embodiment.

FIG. 7 is a diagram illustrating an MPEG2-PS reference decoder.

[Fig. 8] Fig. 8 is a diagram for describing a multiplexing state of packets of video and audio information and packet grouping by packs.

FIG. 9 is a diagram illustrating a data storage state and a read timing of the STD buffer of the decoding device.

FIG. 10 is a diagram showing a configuration of a conventional real-time authoring system.

[Explanation of symbols]

1, 1'Encoding device 3 Video encoder 5 Audio encoder 9, 13, 17 Elementary stream file 11, 15, 19, 47, 51 Encoding additional information file 21, 21-1, 21-2 Multiplexing Device (MUX) 23 Scheduling device 25 Pack information 27 Packet information 29 Reference image position information generator 33 PSD information file 35 PSM information file 37 Multi-bitstream generator 51 Storage medium 55 Separation device (DMUX) 57 ', 59' STD Buffer 57 Video decoder (decoder) 59 Audio decoder (decoder) 60 Decoding device 61 System decoder 63 Picture delay memory 65, 67 Buffer memory 69 Multiplexing circuit

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Katsumi Tahara 6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Within Sony Corporation (56) Reference JP-A-5-236009 (JP, A) JP-A 5-236466 (JP, A) JP-A-6-252876 (JP, A) JP-A-6-343065 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04J 3/00 H04N 7/025 H04N 7/03 H04N 7/035 H04N 7/24

Claims (12)

(57) [Claims] 1. An authoring system that multiplexes each elementary stream obtained by individually coding a plurality of individual source signals in accordance with a digital coding standard with another digital signal sequence to generate a multiplex bitstream. In the above, at the time of encoding, each of the elementary streams is respectively generated as a file, and at the same time, an encoding device that simultaneously generates the associated information of encoding as an intermediate file, the file of each elementary stream and the intermediate file, An authoring system comprising: a multiplexer that schedules multiplexing without analyzing an elementary stream to generate a multiplexed bitstream. 2. The authoring system of claim 1, wherein the digital coding standard is MPE for moving images.
An authoring system that is a G2 system and is MPEG audio for audio. 3. The authoring system according to claim 1, wherein the encoding device receives at least a video source signal and encodes the video source signal, thereby forming a file of an elementary stream of video data and an intermediate file of encoding accompanying information. And a video encoder for generating a plurality of audio source signals, each of which receives and encodes an audio source signal,
An authoring system having a stream file and a plurality of audio encoders for generating an intermediate file of encoded side information. 4. The authoring system according to claim 3, wherein the encoding device further receives and encodes at least one subtitle source signal, and an elementary stream file of subtitle data and an accompanying encoding. An authoring system having a subtitle encoder for generating an intermediate file of information. 5. The authoring system according to claim 1, wherein the multiplexer includes a scheduling unit, a reference image position information generator, and a multiple bit stream generator, and the scheduling unit performs multiplexing. Performing the scheduling of each elementary stream to generate the result as an intermediate file of the schedule data, the reference image position information generator, for each pack constituting a multiple bit stream from the intermediate file of the schedule data An intermediate file of reference image position information to be inserted is generated, and the multiplex bitstream generator generates multiplex bitstreams from each elementary stream based on the information of the intermediate file of the schedule data and the intermediate file of the reference image position information. The census to generate Gushisutemu. 6. The authoring system according to claim 5, wherein the size of each access unit, which is at least a minimum unit of multiplexing, D in the intermediate file of the coded accompanying information.
An authoring system that describes TS and PTS. 7. A multiple bit stream obtained by multiplexing a plurality of individual source signals including at least a video source signal and an audio source signal in accordance with a digital moving image encoding standard and multiplexing the obtained elementary streams. A coding device used in an authoring system for generating a plurality of individual signals, each of which has a plurality of encoders, each encoder compressing and encoding the individual signal An encoding device that generates a file of a stream and a file of corresponding encoding accompanying information. 8. Authoring for multiplexing each elementary stream obtained by encoding a plurality of individual signals including at least a video signal and an audio signal to generate a multiple bit stream according to a digital moving image encoding standard. In the multiplexing device used in the system, a scheduling device that schedules each elementary stream to be multiplexed and generates an intermediate file of schedule data, and an intermediate file of reference image position information from the intermediate file of the schedule data is generated. A reference image position information generator for generating, and a bitstream generator for generating a multiple bitstream from each elementary stream based on information of the intermediate file of the reference image position information and the intermediate file of the schedule data are provided. Multiplexer. 9. An authoring system for encoding an individual stream and multiplexing each encoded bitstream to generate a multiplexed bitstream, wherein the authoring system comprises an encoding device and a multiplexing device, wherein the encoding device is , At the same time as generating an elementary stream as a file at the time of encoding, simultaneously generating encoding accompanying information as an intermediate file, the multiplexing device includes a multiplex scheduling unit, a reference image position information generator, and a multiplex bitstream generator. The multiplexing scheduling device generates multiplexing schedule data as an intermediate file from the intermediate file of the encoding accompanying information without analyzing the elementary stream, and generates the reference image position information. Is the middle of the schedule data Authoring system generates the reference image position information from Airu, the multi-bit stream generator for generating a multi-bit stream intermediate file of the reference position information, from the intermediate file and each elementary stream of the schedule data. 10. A multi-bit by multiplexing each elementary stream obtained by encoding a plurality of individual source signals including a video source signal, an audio source signal, and a caption source signal, and another digital signal sequence. In the method for generating a stream, in the encoding step, each elementary
At the same time that the stream file is generated, each of the coded incidental information is generated as an intermediate file, and in the multiplexing step, the intermediate file is taken in to perform multiplexing scheduling without analyzing the elementary stream. A method of generating a multiplexed bitstream, which is performed and multiplexed. 11. The method of generating a multi-bitstream according to claim 10, wherein the step of multiplexing includes a step of scheduling, a step of generating reference image position information, and a step of generating a multi-bitstream. In the scheduling step, each elementary stream is scheduled based on the intermediate file of the encoding accompanying information, and the result is generated as an intermediate file of schedule data, and the step of generating the reference image position information is performed in the schedule. The step of generating an intermediate file of reference image position information to be inserted for each pack forming a multiplex bitstream from the intermediate file of data and generating the multiplex bitstream includes Intermediate file of information Method for generating a multi-bit stream to generate a multiplexed bit stream from each elementary stream based on information Le. 12. The method for generating a multi-bitstream according to claim 11, wherein in the intermediate file of the coded ancillary information, at least a size of each access unit that is a minimum unit of multiplexing, D
A method for generating a multiple bitstream in which TS and PTS are described.