KR100233937B1 - Video and audio synchronization method using timestamp compensation and mpeg-2 encoder - Google Patents

Abstract

The present invention compensates for the difference in the decoding timestamp between the video and audio of the MPEG-2 encoder and the presentation time stamp through the audio portion of the encoder, thereby transmitting a time stamp for synchronizing video and audio in the decoder The present invention relates to a video and audio synchronization method using compensation and an MPEG-2 encoder apparatus using the compensation method. In the encoder, a time stamp considering a coding and a decoding delay element is inserted in an encoder to perform synchronization between video and audio , HDTVs, video-on-demand services, and MPEG-2 encoders and decoders such as DTV.

Description

Video and audio synchronization method using time stamp compensation and MPEG-2 encoder device using the same

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video and audio synchronizing method using timestamp compensation and an MPEG-2 encoder device using the same. More particularly, the present invention relates to an MPEG- And an MPEG-2 encoder device using the video and audio synchronizing method using timestamp compensation for synchronizing video and audio in a decoder by compensating and transmitting an expression time stamp through the audio portion of the encoder.

In general, the quality of time-dependent multimedia services depends on the continuity of the media objects constituting the service and the synchronization between the media objects. This applies equally to demanding services and isochronous services. Continuity for a media object enables stable reproduction of individual media, which means that the period in which the media object is represented is constant.

In addition, synchronization between time-dependent media objects is possible by identifying the temporal correlation. However, MPEG-2 data generated in an encoder system in which such a temporal correlation has not been determined is difficult to be synchronized in a decoder. In order to compensate for this, a method of adjusting a decoding time or a reproduction time of a medium artificially delayed is used. As an example, when a video frame and an audio frame are transmitted to a decoder, And a decoding time stamp (hereinafter referred to as " DTS ") value in which a reproduction time is set are inserted and transmitted, and a DTS value is inserted in the audio frame So as to perform decoding.

However, this method is also difficult to accurately synchronize video and audio, and is inadequate for real-time service. As a best practice, it is considered to insert an accurate time stamp in the encoder.

In order to solve the conventional problems as described above, the present invention provides a method for synchronizing video and audio by inserting respective time stamps considering coding and decoding delay factors in an encoder into a transmission frame, And an encoder device using the same.

FIG. 1 is a schematic block diagram showing a configuration of a video and audio codec system to which the present invention is applied and a delay model of each component.

FIG. 2 is a block diagram of an MPEG-2 encoder apparatus using a time stamp compensation method according to the present invention; FIG.

Description of the Related Art

11: Video encoder 12: Audio encoder

13: multiplexer 14: demultiplexer

15: video decoder 16: audio decoder

21: video encoder 22: audio encoder

23: Video PES packetizer 24: Audio PES packetizer

25, 26: Receive buffer 27: Multiplexer

28: Timestamp compensator

To achieve the above object, there is provided a method of synchronizing video and audio in an MPEG-2 system, comprising the steps of: receiving a decoding time stamp (DTS) and a presentation time stamp (DTS) inserted in the video frame; and a second step of calculating a value (Dcommpen) for correcting synchronization between the video data and the audio data using the DTS inserted in the video frame; And a third step of inserting DTS and PTS calculated using the calculated correction value D _compen into the audio frame and transmitting the inserted DTS and PTS, thereby establishing synchronization between video and audio at the encoder end and transmitting it to the decoder end .

In order to achieve the above object, a video packet element stream (PES) packetizer for packetizing a coded signal of a video coder, an audio coder, and a video and outputting the coded signal to a transport stream multiplexer, An audio PES packetizer for outputting the audio PES packet to a transport stream multiplexer, a receiving buffer for receiving video data from the video encoder and transmitting the video data to the video PES packetizer, A video encoder unit comprising: a video access unit indication bit indicating time information for inputting a first byte of a video access unit from a video encoder to a reception buffer; Be Coding Start indicating the time information indicating bits and the decoding time stamp difference value between the audio and video, the first byte of an audio access unit indication bit indicating a time information input to the receive buffer of the audio access unit from the audio encoder to the input (D _compen And a time stamp compensator for compensating the difference value to a decoding time stamp of the audio and outputting a decoding time stamp and a presentation time stamp of each of the video and audio to the video and audio PES packetizer side .

The foregoing and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram showing a configuration of a video and audio codec system to which the present invention is applied and a delay model of each component. The MPEG-2 encoder includes a video encoder 11, an audio encoder 12 and a multiplexer 13 . The MPEG-2 decoder is composed of a demultiplexer 14, a video decoder 15 and an audio decoder 16.

Each component of the model has inherent coding, decoding, multiplexing, and demultiplexing delays. La the video and audio encoders, each encoder of the delay d _ve, d _ae, and La, respectively the video and audio decoder in the decoder delay d _vb, d _ad. The delays of the multiplexing unit 13 and the demultiplexing unit 14 are defined as d _mux and d _demux , respectively. The transport stream generated by the encoder is transmitted through the communication network. The delay occurring in the network at this time is referred to as d _net . A description of these delay parameters is as follows.

d _ve is subjected to pre-treatment time and a pre-processing required to create the color signal component to the the analog image signal input from the camera is converted to a digital signal video encoder 11 coding to store the data in the frame memory frame image data Is the time taken until the first compressed image data obtained by encoding the first compressed image data is generated.

d _ae means the time taken for the analog voice signal input from the microphone to be converted into the digital signal and then stored in the memory until the first coded data is generated. The delays in the corresponding decoders are d _vb, d _ad . On the other hand, d _mux is compressed means that it takes time until the encoded video and O to media data are multiplexed transmission, _and,, d _demux the multiplexing unit 13, the data is demultiplexed for each of video and audio decoders (15 transferred from the And 16, respectively.

On the other hand, video has a unique delay element unlike audio, which is delayed by the virtual buffer. The virtual buffer is used to output the output of a video encoder having a variable bit rate output at an arbitrary bit rate, and to output a data after storing a certain amount of data in the buffer, thereby preventing a decoder from overflowing or overflowing . This delay element is defined as d _{vbv_delay} .

These delay factors use a common time index, which is the 33-bit 90Khz counter value generated from the 27Mhz local oscillator. The value of this counter is called a system time clock counter (STC). STC is the time reference in the system decoder.

The DTS and PTS for video and audio are inserted in the transmission frame in the encoder as mentioned above for a moment. DTS is information for managing the decoding time, and PTS is a time stamp indicating information for managing the reproduction output time.

The insertion of these values is performed in a predetermined unit, which is called an access unit (AU), and a frame is an example of this. Therefore, the insertion unit of the video can be one frame image, and in the case of audio, it is called an audio frame.

The DTS for the kth AU input from the camera is given by the following equation.

_{DTS video (k) = E video} (k) d ve (k) + d vbv_delay (k) + d net k ≥ 0 (1)

In this equation, E _video (k) is the STC value at the time when the kth video is input. Equation 1 means it is the k-th image is input to go through the pre-treatment process of a delay until the first video data is generated (d _ve (k)) and the virtual buffer delay (d _{vbv_delay} (k)), and network transmission delay (d _net ) must be taken into account in order to find the correct DTS. Unlike the DTS, which has a value varying according to the coding mode and the buffer state of the image, the PTS indicating the time at which the image is represented in the decoder can be expressed by the following equation.

In this equation d _ve (k) is the time it takes to represent the storage of a compressed image sent to the frame memory, restore the image data by the video decoder to start decoding. MPEG-2 video is divided into an intra (I) video, a predictive (P) video, a bidirectionally interpolated (B) video referring to an I (or P) video and a P video. Depending on the presence or absence of the decoder, a frame rearrangement occurs in the decoder. Equation 2 is the PTS considering this and a is the difference between decoding time and playback time due to this frame rearrangement. If the coding since they do not have the B-like mode, and is _{α = 0, d vd (k} ) is ignored because it is relatively small when the _video value of PTS (k) = DTS _video (k).

Audio, like video, defines DTS and PTS as follows:

DTS _audio (n) = E _audio (n) + d _ae (n) + d _net n? 0 (3)

PTS _audio = DTS _audio (n) + d _ad (n)

= E _audio (n) + d _ae (n) + d _ad (n) + d _net (4)

In Equations 3 and 4, E _audio (n) is the STC value at the time when the nth audio AU is input, and d _ae (n) is the input of the n th audio and outputs the data of the first compressed audio AU Time. d _ae (n) represents the time required for the compressed and transmitted audio data to be restored and output. The PTS and DTS of audio are closely related to the sampling frequency used in the audio codec and thus have different values. Here, the network delay d _net has the same value as in the case of video.

As described above, the coding and decoding delay of each media is a consideration in the synchronization between media. Generally, the coding and decoding delay of video in MPEG-2 codec system is larger than that of audio, and it changes according to the number of B images which are bidirectional prediction mode. Therefore, in order to synchronize the media with each other in the decoder, it is necessary to compensate the coding and decoding delay difference by adding a certain value to the DTS and PTS of the audio, and the compensated value should be maintained until the end of the service.

The DTS of video and audio analyzed in the present invention can synchronize media objects. Since the coding and decoding delay of audio is relatively small compared to video, if the value of the first delay difference is corrected, You can synchronize. This correction value is the difference for the initial video and audio DTS, and each initial delay is as follows.

_{DTS video (0) = E video} (0) + d ve (0) + d vbv_delay (0) + d net (5)

DTS _audio (0) = E _video (0) + d _ae (0) + d _net (6)

In Eqs. 5 and 6, E _video (0) and E _audio (0), which are the initial input points of each media, have the same STC value since they are at the same time. Therefore, DTS _{compen, which} is the compensation value of audio DTS for synchronizing the media, is expressed by the following equation

DTS _{compen =} DTS _video (0) - DTS _audio (0)

_{= D ve (0) + d} vbv_delay (0) -d ae (0) (7)

Therefore, when the audio is served as video, the following values Lt; / RTI >

Through the above quantitative analysis, the time stamp correction value of the audio for synchronizing the complete video and audio was examined. Here, there is an unknown delay element without the encoder providing the information, which includes the decoding delay elements d _vd (0), d _ad (0) and d _net of the decoder. These delay factors are the values that can be ignored when compared to the total delay factor.

2 shows a configuration of an MPEG-2 encoder apparatus using timestamp compensation in the case where d _vd (0), d _ad (0), and d _net delay elements are excluded in the present invention. The conventional encoder apparatus includes a video encoder 21, an audio encoder 22, a packetized elementary stream (hereinafter, referred to as PES) packetized by packetizing a coded video signal and outputting the packetized signal to a transport stream multiplexer 27 An audio PES packetizer 24 for packetizing an audio encoded signal and outputting the packetized audio signal to a transport stream multiplexer 27 and a video encoder 24 for receiving the video data from the video encoder 21, A receiving buffer 25 for transmitting the audio data to the video PES packetizer 23 and a receiving buffer 26 for receiving audio data from the audio encoder 22 and transmitting the audio data to the audio PES packetizer 24 .

In the present invention, the compensation value D _compen is obtained in the encoder device, A time stamp compensator 28 for generating a value is introduced.

When the video encoder 21 notifies the 90-kHz counter and the time stamp compensator 28 of the time information indicating the time at which the frame image is input from the camera, the value of the 90-kHz counter is set by the compensator 28 using this information. This value becomes E _video (0). At this time it starts to convert the audio data _{A / D. (E audio (} 0). When the first byte, the time information of video AU indication bit which is input to the reception buffer 25 from the time the video AU is input to the compensator d _vde (0) it is calculated, as is also the audio d _ae (0) is calculated. d _{vbv_delay} (0) is that the time required until an amount video data header of the first is defined in the virtual buffer to accumulate in the video data It is possible to obtain the D _compen value, which is the compensation value given in Equation 7. Thus, the DTS and the PTS of the compensated audio can be obtained by using this value , The decoder can reproduce video and audio synchronized with each other.

As described above, according to the present invention, the DTS and the PTS of the video and audio, which indicate the point in time at which the AU should be decoded, are precisely obtained in the encoder and the synchronization of video and audio is compensated so that the decoder can synchronize the video and the audio of the decoder , Which has the advantage of being widely used in devices that use MPEG-2 encoders and decoders such as HDTVs, video-on-demand services, and DTVs.

It will be apparent to those skilled in the art that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. You should see.

Claims (8)

In the MPEG-2 system, when a data frame is transmitted from an encoder end to a decoder end, a decoding time stamp (DPS) and a presentation time stamp (PTS) are inserted in a video frame for transmission A first step of: A second step of calculating a value (D _compen ) for correcting the synchronization between the video data and the audio data using the DTS inserted in the video frame; And a third step of inserting DTS and PTS calculated using the calculated correction value D _compen into the audio frame and transmitting the inserted DTS and PTS, thereby establishing synchronization between video and audio at the encoder end and transmitting the same to the decoder end A method for synchronizing video and audio using time stamp compensation. The method of claim 1, wherein the correction value (D _compen ) calculated in the second step is calculated through the following equation. _{D comper = d ve (0)} + d vbv_delay (0) -d ae (0) (d _ve (0): the first image input is undergoing a pre-processing delay until the first video data is generated d _{vbv_delay} (0): Virtual buffer delay for the first video input d _ae (0): the time required for the first compressed audio access unit (AU) to output the first audio input A video encoder, a video encoder, a video packet element (PES) packetizer for packetizing the encoded signals of the video and outputting them to a transport stream multiplexer, and outputting the packetized audio signals to a transport stream multiplexer A receiving buffer for receiving video data from the video encoder and transmitting the video data to the video PES packetizer, and a receiving buffer for receiving audio data from the audio encoder and transmitting the audio data to the audio PES packetizer A video encoder unit comprising: a video access unit indication bit indicating time information that a first byte of a video access unit is input to a receive buffer from the video encoder; Due to the audio access unit to input an instruction bits and the coded bits representing the time information is input to the buffer calculating the decoding time stamp (DTS) difference of the audio and video (D _compen), and the difference value to the decoding time stamp of the audio And outputting a decoding time stamp and a presentation time stamp (PTS) of each video and audio to the video and audio PES packetizer by compensating the MPEG- ) Encoder device. The method of claim 1, wherein the DTS calculated in the third step is calculated through the following equation: _{DTS audio (n) = E audio} (n) + d ae (n) -d ae (0) + d ve (0) + d vbv_delay (0) + d net (N≥0, _audio E (n): n-th audio access systems, a clock counter (STC) value in the unit time of the input, d _ae (n): n-th input speech is first compressed audio access unit ( AU) time there is a data output, d _ae (0): the first audio is input time it takes to be first compressed audio access units coming from the data output, d _ve (0): first image D _{vbv_delay} (0): delay in the virtual buffer for the first video input, d _net : delay occurring in the network) The method of claim 1, wherein the PTS calculated in the third step is calculated through the following equation. PTS _audio (n) = DTS _audio (n) + d _ad (n) in the compensated audio frame (n? 0, d _ad (n): time required for restoring and outputting compressed audio data) 4. The apparatus of claim 3, wherein the decoding time stamp difference value (D _compen ) of the audio and the video is calculated by the following equation. _{D compen = d ve (0)} + d vbv_delay (0) - d ae (0) (D _ve (0): the first image is input to go through the pre-processing delay until the first video data is generated, d _{vbv_delay} (0): the virtual buffer for a first image input delay, d _ae (0): The time required for inputting the first voice and outputting the data of the first compressed audio access unit AU) 4. The apparatus of claim 3, wherein the audio DPS output from the time stamp compensator is calculated by the following equation. _{DTS audio (n) = E audio} (n) + d ae (n) - d ae (n) + d ve (0) + d vbv_delay (0) + d net (n): e _audio (n): the system clock counter (STC) value at the time when the nth audio ACCESS UNIT is input, _DAE (n): the nth audio is input and the first compressed audio ACCESS UNIT AU) time there is a data output, d _ae (n): the time the maximum set of speech is input required to the data in the first compressed audio access unit is output, d _ve (0): the first image D _{vbv_delay} (0): false buffer delay for the first image input, d _net : delay occurring in the network) The MPEG-2 encoder apparatus according to claim 3, wherein the audio PTS output from the time stamp compensator is calculated by the following equation. PTS _audio (n) = DTS _audio (n) output from the time stamp compensator + d _ad (n) (n? 0, d _ad (n): time required for restoring and outputting compressed audio data)