JPH11234234A - Synchronous multimedia multiplexing method and encoder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the software processing of synchronous multiplexing using a general-purpose computer, which is operated by any arbitrary clock independent of a time reference clock, or the distributed cooperative processing of an encoder due to plural computers to refer to different clocks. SOLUTION: An audio encoding part 11a and a video encoding part 11v output an audio event notice 23 and a video event notice 23v and a time reference event detecting part 11r outputs a time reference event notice 23r to an interruption generating part 12. The interruption generating part 12 generates an interruption 24 together with the kind of the event notice received at a synchronous multiplexing part 2. A clock counter 14 counts a reference clock 13. The synchronous multiplexing part 2 discriminates the kind of the event, which occurs from the interruption 24, and the occurrence time of the relevant event is converted to a media synchronizing signal and multiplexed to a multimedia stream 27.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a synchronous multiplexing method and an encoder for transmitting multimedia information such as audio and video when transmitting information.

[0002]

2. Description of the Related Art FIG. 7 is a block diagram of a general multimedia information transmission system.

[0003] The multimedia transmission system comprises an encoder 46 and a decoder 49. Encoder 46
Is an audio signal 43 input from the audio input device 41,
The video signal 44 input from the video input device 42 and other user-defined signals 45 defined by the user are encoded, synchronously multiplexed into a single multimedia stream 47, and transferred to a transmission medium 48 such as a transmission line or an information storage device. /accumulate. The decoder 49 separates and decodes the multimedia stream 47 from the transmission medium 48, and
0, the decoded video signal 51, and the decoded user data 52 are reproduced by the speaker 53 and the display 54 in accordance with the synchronization relationship intended on the encoder 46 side.

Next, prior to the description of the conventional technology, a multimedia synchronous multiplexing system using a time reference signal and a time stamp specified in "ISO / IEC 13818-1 MPEG2 System" will be described with reference to FIG.

[0005] In FIG. 8, reference numeral 55 denotes a natural time axis representing natural time, and reference numeral 56 denotes a time reference clock of the encoder 46 (hereinafter simply referred to as a reference clock).

In the multimedia transmission system, a reference clock 56 of the same frequency is supplied to both the encoder 46 and the decoder 49, and the encoder 46 and the decoder 49 individually recognize the system time by counting the reference clock 56. In FIG. 8, the number attached to the time reference clock waveform is a time counter value, and is simply referred to as time hereinafter.

[0007] In the multimedia transmission system, the audio signal 43 and the video signal 44 are divided into audio events 57a and video events 57v, which are units of encoding, decoding and synchronous multiplexing, respectively. The data is encoded by the multiplexing unit 62v, synchronously multiplexed by the synchronous multiplexing unit 63, and transmitted to the decoder 49. Event 57a, 5
7v is called an access unit in the "ISO / IEC 13818-1 MPEG2 System", defines an audio access unit by a sequence of a fixed number (for example, 384, 1152) of continuous sampled audio, and a video access unit forms a video. Defined in the picture.

In the multimedia transmission system, the encoder 46 sets the value of the clock counter 64 to an intended value, and performs “clock adjustment” between the encoder 46 and the decoder 49.

The encoder 46 generates a time reference event 57r in accordance with a predetermined rule, generates time reference information 58 from the occurrence time (t1 in FIG. 8) of the time reference event 57r, and multiplexes it on the multimedia stream 47. I do. In the example of FIG. 8, the time reference event occurrence time is used as the time reference information 58 as it is.

Upon reading the time information reference signal 59, the decoder 49 immediately sets the time information reference signal 59 (t1 in FIG. 8) in the clock counter 64 and continues incrementing the clock counter 64.

The multimedia transmission system causes the decoder 49 to reproduce events according to the synchronization relationship intended by the encoder 46 in accordance with the following procedure.

If the event 57 occurs, the encoder 46 detects the start time of the event 57, and generates a time stamp 59 from the event occurrence time (in FIG. 8, the audio event 57a is t4 and the video event 57v is t5).
Multimedia stream 4 with encoding event 60
7 is multiplexed. In FIG. 8, the time reference event occurrence time is used as it is as the time stamp 59.

The decoder 49 detects the time stamp 59 and reproduces the encoded event 61 when the value of the clock counter 64 matches the time stamp 60.

With the above procedure, the multimedia transmission system can reproduce the decoded audio 61a and the decoded video 61v on the decoder 49 side according to the synchronization relationship intended by the encoder 46.

In FIG. 8, for simplicity, only audio and video are limited, and only events and time stamps are limited to reproduction. "ISO / IEC 13818-1 MPE
In the “G2 System”, the definition of the event is not limited to the example in FIG. 8 as the decoding time stamp is specified as the time stamp.

Referring to FIG. 9, the configuration of an encoder 46 for performing conventional multimedia synchronous multiplexing will be described.

In the conventional encoder 46, the reference clock 56 is input to the synchronous multiplexing section 63, and a clock counter 64 is installed in the synchronous multiplexing section 63 to count time.
The encoding unit 62 detects the occurrence of the event 57 and issues an event occurrence notification 67 (67a, 67v) to the synchronous multiplexing unit 63. The synchronous multiplexing unit 63 receives the event occurrence notification 67 and multiplexes the time information signal 58 and the time stamp 59 into the multimedia stream 47 according to the event 57 type.

GLOBCOM '95, 13.4 "MPEG2 Video and Audio CODEC"
Board Set for Personal Computer "and" MPEG2 Video and Audio CODEC Board Set f
or Personal Computer "as a multiplex in The Eur
opean Design and Test Conference (ED & TC) 1996 "A
Memory-based Aechitecture for MPEG2 System Protoc
ol LSIs ", all of which are handled by dedicated hardware.

[0019]

In the above-mentioned conventional decoder, the synchronous multiplexing unit uses a general-purpose computer operating with an arbitrary clock independent of the time reference clock because the synchronous multiplexing unit measures the time by referring to the time reference clock. This has hindered the software processing of synchronous multiplexing and the distributed cooperative processing of encoders by multiple computers that refer to different clocks.

An object of the present invention is to provide software processing for synchronous multiplexing using a general-purpose computer operating at an arbitrary clock independent of a time reference clock, and distributed cooperative processing of an encoder by a plurality of computers referencing different clocks. It is an object of the present invention to provide a multimedia synchronous multiplexing and encoder capable of realizing the above.

[0021]

According to the multimedia synchronous multiplexing method of the present invention, the encoding step detects the occurrence of an event and generates an interrupt to the synchronous multiplexing step.
In the synchronous multiplexing step, the type of the event that has occurred from the interrupt is determined, the event occurrence time of the event is referred to, and the event occurrence time is determined by a media synchronization signal (video stream). Multiplexed into a multimedia stream after conversion into a signal of a format defined as the syntax of

By eliminating the supply of the time reference clock to the multiplexing stage, software processing by a synchronous multiplex general-purpose computer and distributed cooperative processing by a plurality of computers having different clocks become possible.

[0023]

Next, embodiments of the present invention will be described with reference to the drawings.

Referring to FIG. 1, an encoder according to a first embodiment of the present invention includes an audio encoding unit 11a and a video encoding unit 1
1v, a time reference event detecting section 11r, an interrupt generating section 12, an encoding section 1 including a clock counter 14 for counting a reference clock 13, and a synchronous multiplexing section 2.

Audio encoding section 11a, video encoding section 11v
Encodes the audio signal 21a and the video signal 21v, respectively, outputs the audio coded signal 22a and the video coded signal 22v, and outputs the audio event notification 23a and the video event notification 23v to the interrupt generation unit 12. The time reference event detection unit 11r detects a time reference event 21r output when performing “time adjustment” by generating time reference information, and outputs a time reference event notification 23r to the interrupt generation unit 12. The interrupt generation unit 12 outputs the audio event interrupt notification 23a and the video event notification 23
v, When the time reference event notification 23r is received, an interrupt 24 corresponding to the type of the event notification received by the synchronous multiplexing unit 2 is generated. The clock counter 14 counts the reference clock 13 and transfers the count value (event occurrence time) to the synchronous multiplexing unit 2.

If the synchronous multiplexing unit 2 receives the interrupt 24, it determines the type of the interrupt. If the interrupt is a time reference event interrupt, it generates time reference information from the value of the clock counter 14 and multiplexes it into the multimedia stream 27. If an audio event interrupt occurs, an audio time stamp which is a media synchronization signal is generated, and if a video event interrupt occurs, a video time stamp which is a media synchronization signal is generated from the value of the clock counter 14 to generate a multimedia stream 27.
Multiplex.

Referring to FIG. 2, the encoder according to the second embodiment of the present invention includes an audio event time buffer 15a, a video event time buffer 15v, and a time buffer in the encoder 1 of the encoder according to the first embodiment of FIG. It has a reference event time buffer 15r. The audio event time buffer 15a, the video event time buffer 15v, and the time reference event time buffer 15r output the clock counter 14 when the audio event notification 23a, the video event notification 23v, and the time reference event notification 23r are output, respectively.
Captures the current value of

When the synchronous multiplexing unit 2 receives the interrupt 24 from the encoding unit 1 as in the first embodiment, the synchronous multiplexing unit 2 determines the type of the interrupt, reads the value of the corresponding event time buffer (event occurrence time), This is converted into a multimedia synchronization signal and multiplexed into a multimedia stream 27.

Referring to FIG. 3, the encoder according to the third embodiment of the present invention differs from the first embodiment in FIG. 1 in that the synchronous multiplexing section 2 has an address generation section 16 and the encoding section 1 has an address decoder 17A. It is provided with.

When the synchronous multiplexing unit 2 receives the interrupt 24 from the encoding unit 1, the address generating unit 16 determines the type of the interrupt, generates the corresponding read address 26 of the clock counter 14 (plural), and Output to 17A. The address decoder 17A has a read address 2
6 is decoded, a read enable signal 28 is output to the clock counter 14, and the value is read from the clock counter 14. Other operations are the same as in the first embodiment.

Referring to FIG. 4, an encoder according to a fourth embodiment of the present invention comprises an address generator 16 in the synchronous multiplexing unit 2 and an address decoder in the encoder 1 of the encoder according to the second embodiment in FIG. 17B.

When the address decoder 17B generates and outputs the read address 26 from the address generator 16, it decodes the read address 26 and outputs a read enable signal 28 to the corresponding time reference event buffer.
Other operations are the same as in the fourth embodiment.

Referring to FIG. 5, the encoder according to the fifth embodiment of the present invention is different from the encoder according to the fourth embodiment in FIG. 4 in that an audio event time buffer 15a, a video event time buffer 15v, and a time reference event time buffer are provided. Fifteen
r is selected according to the read enable signal 28,
It is provided with a selecting section 18 for outputting to the synchronous multiplexing section 2.

Referring to FIG. 6, a multi-encoder according to a sixth embodiment of the present invention comprises a sound event time buffer 1
5a, the video event time buffer 15v, and the time reference event time buffer 15r are mapped as agents of the general-purpose computer 31 and the bus 36, and the general-purpose computer 31 performs synchronous multiplexing while referring to the event time buffers 15a to 15r. Of the software 32.

[0035]

As described above, according to the present invention, there are provided: 1. Real-time processing of multimedia multiplexing is possible with software of a general-purpose computer that operates at an arbitrary system clock. The effect of real-time processing of multimedia multiplexing by distributed cooperative processing of a plurality of general-purpose computers operating at an arbitrary system clock is obtained.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an encoder according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram of an encoder according to a second embodiment of the present invention.

FIG. 3 is a configuration diagram of an encoder according to a third embodiment of the present invention.

FIG. 4 is a configuration diagram of an encoder according to a fourth embodiment of the present invention.

FIG. 5 is a configuration diagram of an encoder according to a fifth embodiment of the present invention.

FIG. 6 is a configuration diagram of an encoder according to a sixth embodiment of the present invention.

FIG. 7 is a configuration diagram of a multimedia information transmission system.

FIG. 8 is a configuration diagram of a multimedia information transmission system in MPEG2.

FIG. 9 is a configuration diagram of a conventional encoder.

[Explanation of symbols]

 Reference Signs List 1 encoding unit 2 synchronous multiplexing unit 11a audio encoding unit 11v video encoding unit 11r time reference event detection unit 12 interrupt generation unit 13 reference clock 14 clock counter 15a audio event time buffer 15v video event time buffer 15r reference time event time Buffer 16 Address generator 17A, 17B Address decoder 18 Selector 21a Audio signal 21v Video signal 21r Time reference event 22a Encoded audio signal 22v Encoded video signal 23a Audio event notification 23v Video event notification 23r Time reference event notification 24 Interrupt 26 Read Address 27 Multimedia stream 28 Read enable signal 30 Bus 31 General-purpose computer 32 Synchronous multiplexing software

──────────────────────────────────────────────────の Continued on front page (51) Int.Cl. ⁶ Identification code FI H04N 7/24

Claims (9)

[Claims] 1. A multimedia synchronous multiplexing step having a coding step of coding a multimedia signal and a synchronous multiplexing step of multiplexing the coded multimedia signal while maintaining mutual synchronization and outputting as a multimedia stream. The method, wherein the encoding step comprises detecting an occurrence of an event, generating an interrupt to the synchronous multiplexing step, and measuring an event occurrence time, wherein the synchronous multiplexing step comprises generating the event from the interrupt. Multimedia type synchronous multiplexing, comprising the steps of: determining the type of the event, referring to the event occurrence time of the event, converting the event occurrence time into a media synchronization signal, and multiplexing the multimedia stream into a multimedia stream. Method. 2. The method of claim 1, wherein said encoding step comprises the step of transferring a measured event occurrence time to said synchronous multiplexing step. 3. The method according to claim 1, wherein said encoding step includes storing the measured event occurrence time in a storage unit. 4. An audio encoding unit that encodes an audio signal and generates an audio event notification, a video encoding unit that encodes a video signal and generates a video event notification, and detects a time reference event, A time reference event detection unit that generates a time reference event notification; an interrupt generation unit that generates an interrupt according to the received event notification when receiving the audio event notification, the video event notification, and the time reference event notification; An encoding unit including a clock counter that counts a clock and outputs an event occurrence time, and generates a multimedia stream from the encoded audio signal and the encoded video signal output from the encoding unit, Determines the type, and if an audio event interrupt occurs, an audio timestamp and video event interrupt A multiplexing unit that generates a video time stamp from the event occurrence time and multiplexes the multimedia stream with the multimedia stream. Encoder having 5. An audio encoding unit for encoding an audio signal and generating an audio event notification, a video encoding unit for encoding a video signal and generating a video event notification, detecting a time reference event, A time reference event detection unit that generates a time reference event notification; an interrupt generation unit that generates an interrupt according to the received event notification when receiving the audio event notification, the video event notification, and the time reference event notification; A clock counter that counts clocks and measures the event occurrence time;
When the audio event notification, the video event notification, and the time reference event notification occur, an encoding unit including an audio event time buffer, a video event time buffer, and a time reference event time buffer for storing the event occurrence time at that time. And, while generating a multimedia stream from the encoded audio signal and the encoded video signal output from the audio encoding unit, determine the type of the interrupt, the audio event time buffer, the video event time buffer,
The event occurrence time is read from the corresponding buffer among the time reference event time buffers, and an audio time stamp is generated from the event occurrence time if the event is an audio event interrupt, and a video time stamp is generated from the event occurrence time if the event is a video event interrupt. An encoder having a multiplexing unit that multiplexes and generates time reference information from the event occurrence time if the notification is a time reference event notification, and multiplexes the time reference information with the multimedia stream. 6. An audio encoding unit for encoding an audio signal and generating an audio event notification, a video encoding unit for encoding a video signal and generating a video event notification, detecting a time reference event, A time reference event detection unit that generates a time reference event notification; an interrupt generation unit that generates an interrupt according to the received event notification when receiving the audio event notification, the video event notification, and the time reference event notification; An encoding unit including an address decoder that decodes the read address obtained and outputs a read enable signal to the clock counter; and an address generation unit that determines the type of the interrupt and generates the read address. A multimedia stream from the encoded audio signal output from the encoding unit and the encoded video signal. It generates the over arm, type the event occurrence time output from the clock counter,
In the case of an audio event interrupt, an audio time stamp is generated, and in the case of a video event interrupt, a video time stamp is generated from the event occurrence time and multiplexed into the multimedia stream. An encoder having a multiplexing unit for generating and multiplexing the multimedia stream with the multimedia stream. 7. An audio encoding unit that encodes an audio signal and generates an audio event notification, a video encoding unit that encodes a video signal and generates a video event notification, and detects a time reference event, A time reference event detection unit that generates a time reference event notification; an interrupt generation unit that generates an interrupt according to the received event notification when receiving the audio event notification, the video event notification, and the time reference event notification; A clock counter that counts clocks and measures the event occurrence time;
When the audio event notification, the video event notification, and the time reference event notification occur, respectively, an audio event time buffer storing the event occurrence time at that time, a video event time buffer, and a time reference event time buffer are input. An encoding unit having an address decoder for decoding a read address and outputting a read enable signal to a corresponding buffer of the audio event time buffer and the video event time buffer; determining an interrupt type; Including an address generation unit for generating and outputting to the encoding unit;
A multimedia stream is generated from the encoded audio signal output from the audio encoding unit and the encoded video signal, and an event occurrence time output from the clock counter is input. In the case of a video event interrupt, a video time stamp is generated from the event occurrence time and multiplexed with the multimedia stream, and if a time reference event notification, time reference information is generated from the event occurrence time and the multimedia stream is generated. An encoder having a multiplexing unit for multiplexing. 8. An audio encoding unit for encoding an audio signal and generating an audio event notification, a video encoding unit for encoding a video signal and generating a video event notification, detecting a time reference event, A time reference event detection unit that generates a time reference event notification; an interrupt generation unit that generates an interrupt according to the received event notification when receiving the audio event notification, the video event notification, and the time reference event notification; A clock counter that counts clocks and measures the event occurrence time;
When the audio event notification, the video event notification, and the time reference event notification occur, respectively, an audio event time buffer, a video event time buffer, and a time reference event time buffer storing the event occurrence time at that time; A selection unit that selects an output corresponding to a read enable signal among outputs of a time buffer, the video event time buffer, and the time reference event time buffer; and a read address, and the audio event time buffer and the video event time buffer An encoding unit that includes an address decoder that outputs the read enable signal to a corresponding buffer among the time reference information event time buffers; and an address generation unit that determines the type of the interrupt and generates the read address. Said A multimedia stream is generated from the encoded audio signal output from the audio encoding unit and the encoded video signal, and the event occurrence time output from the selection unit is input. In the case of a video event interrupt, a video time stamp is generated from the event occurrence time and multiplexed with the multimedia stream. Encoder having a multiplexing unit for converting. 9. The encoder according to claim 5, wherein said synchronous multiplexing unit comprises a general-purpose computer connected to said encoding unit via a bus.