KR19980027646A - Video and Audio Synchronization Method Using Timestamp Compensation and MPEG-2 Encoder Device Using It - Google Patents

Abstract

The present invention relates to a video and audio synchronizing method using time stamp compensation and an MPEG-2 encoder apparatus using the same, and more particularly, to a video and audio synchronizing method using time stamp compensation, And an MPEG-2 encoder using the time stamp compensation for audio synchronization.

The present invention provides a method for synchronizing video, which is a medium, with audio by inserting a time stamp considering coding and decoding delay factors in an encoder without any operation in the decoder, and an encoder device using the method.

According to another aspect of the present invention, there is provided an apparatus and method for compensating for a difference in decoding timestamp between video and audio in an MPEG-2 encoder, An audio synchronization method and an MPEG-2 encoder device using the same are provided.

Description

Video and Audio Synchronization Method Using Timestamp Compensation and MPEG-2 Encoder Device Using It

FIG. 1 is a diagram illustrating a configuration of a general video and audio codec system and a delay model of each component;

FIG. 2 is an MPEG-2 encoder apparatus using the time stamp compensation method according to the present invention.

Description of the Related Art [0002]

11: Video encoder 12: Audio encoder

13: multiplexer 14: demultiplexer

15: video decoder 16: audio decoder

21: video encoder 22: audio encoder

23: Video PES packetizer 24: Audio PES packetizer

25, 26: Receive buffer 27: Multiplexer

28: Timestamp compensator

The present invention relates to a video and audio synchronizing method using time stamp compensation and an MPEG-2 encoder apparatus using the same, and more particularly, to a video and audio synchronizing method using time stamp compensation, And an MPEG-2 encoder using the time stamp compensation for audio synchronization.

The quality of time-dependent multimedia services depends on the continuity of the media objects that make up the service and the synchronization between the media objects. This applies equally to demanding services and isochronous services. The continuity of the media object enables stable reproduction of the individual media, which means that the period in which the media object is represented is constant. In addition, synchronization between time-dependent media objects is possible by identifying temporal correlation. However, it is difficult to synchronize MPEG-2 data generated in an encoder system that has no correlation with time in the decoder. In this case, a method of adjusting the decoding time or reproduction time of the artificially delayed media is used . However, this method is inappropriate for real-time services, and the best method is to insert the correct time stamp in the encoder.

Therefore, in order to solve the above problems, the present invention provides a method for synchronizing video and audio, which is a medium, by inserting a time stamp considering coding and decoding delay factors in an encoder without any operation in a decoder and an encoder device using the method I want to.

According to another aspect of the present invention, there is provided an apparatus and method for compensating for a difference in decoding timestamp between video and audio in an MPEG-2 encoder, An audio synchronization method and an MPEG-2 encoder device using the same are provided.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features, and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 shows the configuration of a general video and audio codec system and a delay model of each component. The MPEG-2 encoder is composed of a video encoder 11, an audio encoder 12 and a multiplexer 13. The MPEG-2 decoder is composed of a demultiplexer 14, a video decoder 15 and an audio decoder 16. Each component of the model has inherent coding, decoding, multiplexing, and demultiplexing delays. Let the video and audio encoders of the encoder is delayed _ve d, d _ae la, each of the video and audio decoder in the decoder delay d _vd, d _ad d respectively. The delays of the multiplexer 13 and the demultiplexer 14 are defined as d _mux and d _demux , respectively. The transport stream generated by the encoder is transmitted through the communication network. The delay occurring in the network at this time is referred to as d _net . A description of these delay parameters is as follows.

d _ve is subjected to pre-treatment time, pre-treatment required is the analog image signal input from the camera converted to a digital signal created by the color signal component to the video encoder 11 coding to store the data in the frame memory frame image data Is the time taken for the first compressed image data to be encoded.

d _ae is the time taken for the analog data input from the microphone to be converted into a digital signal and then stored in the memory until the first coded data is generated. The delays in the corresponding decoders are d _vd and d _ad , respectively. In the meantime, _dmux denotes a time required for multiplexing and transmitting compression-coded video and audio data. In the d _demux , data transmitted from the multiplexing unit 13 is demultiplexed and transmitted to the video and audio decoders 15 and 16 To the time it is transmitted.

On the other hand, video has a unique delay element unlike audio, which is delayed by the virtual buffer. A virtual buffer is used to output the output of a video encoder with a variable bit rate output at an arbitrary bit rate, and to store a certain amount of data in this buffer and then output data to prevent the decoder from overflowing or buffering the buffer . This delay element is defined as d _{vbv_delay} .

These delay factors use a common time index, which is a 33-bit, 90 KHz counter value generated from a 27 MHz local oscillator. The value of this counter is called a system time clock counter (STC). STC is the time reference in the system decoder.

A Decoding Time Stamp (DTS) and a Presentation Time Stamp (PTS) for video and audio are inserted in the encoder. DTS is information for managing the decoding time, and PTS is a time stamp indicating information for managing the reproduction output time.

The insertion of these values is performed in a predetermined unit, which is referred to as an access unit (AU), and a frame is an example thereof. Therefore, the insertion unit of the video can be one frame image, and in the case of audio, it is called an audio frame.

The DTS for the kth AU input from the camera is given by the following equation.

_{DTS video (k) = E video} (k) + d ve (k) + d vbv_delay (k) + d net, k≥0 (1)

In this equation, E _video (k) is the STC value at the time when the kth video is input. Equation 1 means it is the k-th image is input to go through the pre-treatment process of a delay until the first video data is generated (d _ve (k)) and the virtual buffer delay (d _{vbv_delay} (k)), and network transmission delay (d _net ) must be taken into account in order to find the correct DTS. Unlike the DTS, which has a value varying according to the coding mode and the buffer state of the image, the PTS indicating the time at which the image is represented in the decoder can be expressed by the following equation.

In this equation d _ve (k) is the time it takes to represent the storage of a compressed image sent to the frame memory, restore the image data by the video decoder to start decoding. MPEG-2 video is divided into an intra (I) video, a predictive (P) video, a bidirectionally interpolated (B) video referring to an I (or P) video and a P video. Depending on the presence or absence of the decoder, a frame rearrangement occurs in the decoder. Equation 2 is the PTS considering this, and α means the difference between the decoding time and the reproduction time due to the frame rearrangement. If the coding since they do not have the B mode image and is _{α = 0, d ve (k} ) is ignored because it is relatively small when the _video value of PTS (k) = DTS _video (k).

Audio, like video, defines DTS and PTS as follows:

DTS _audio (n) = E _audio (n) + d _ae (n) + d _net , n?

PTS _audio (n) = DTS _audio (n) + d _ad (n)

= E _audio (n) + d _ae (n) + d _ad (n) + d _net (4)

In Eqs. 3 and 4, E _audio (n) is the STC value at the time when the nth audio AU is input, and d _ae (n) It is time. d _ad (n) represents the time taken for the compressed and transmitted audio data to be restored and output. The PTS and DTS of the audio are closely related to the sampling frequency used in the audio codec and thus have different values. Here, the network delay d _net has the same value as in the case of video.

As described above, the coding and decoding delay of each media is a consideration in the synchronization between media. Generally, the coding and decoding delay of video in MPEG-2 codec system is larger than that of audio, and it changes according to the number of B images which are bidirectional prediction mode. Therefore, in order to synchronize the media with each other in the decoder, it is necessary to compensate the coding and decoding delay difference by adding a certain value to the DTS and PTS of the audio, and the compensated value should be maintained until the end of the service.

Using the video and audio DTS analyzed in the present invention, the synchronization between the media objects can be synchronized, which is relatively small compared to the video coding and decoding delay of the audio. Therefore, if the value of the first delay difference is corrected, You can synchronize. This correction value is the difference for the initial video and audio DTS, and each initial delay is as follows.

_{DTS video (0) = E video} (0) + d ve (0) + d vbv_delay (0) + d net (5)

DTS _audio (0) = E _audio (0) + d _ae (0) + d _net (6)

In Eqs. 5 and 6, E _video (0) and E _audio (0), which are the initial input points of each media, have the same STC value since they are at the same time. Therefore, DTS _{compen, which} is the compensation value of the audio DTS for synchronizing the media, is expressed by the following equation.

DTS _compen = DTS _video (0) -DTS _audio (0)

_{= D ve (0) + d} vbv_delay (0) -d ae (0) (7)

Therefore, when the audio is served as video, the following values , Lt; / RTI >

_{audio audio compen}

= E _audio (n) + d _ae (n) + d _net + D _compen (8)

_{= E audio (n) + d} ae (n) -d ae (0) + d ve (0) + d vbv_delay (0) + d net, n≥0

_{audio audio ad} (9)

Through the above quantitative analysis, the time stamp correction value of the audio for synchronizing the complete video and audio was examined. Here, there is an unknown delay element without the encoder providing the information, which includes the decoding delay elements d _vd (0), d _ad (0) and d _net of the decoder. These delay factors are the values that can be ignored when compared to the total delay factor.

FIG. 2 shows a configuration of an MPEG-2 encoder apparatus using timestamp compensation in the case where d _vd (0), d _ad (0) and d _net delay elements are excluded in the present invention. The conventional encoder apparatus includes a video encoder 21, an audio encoder 22, a video PES packetizer 23 for packetizing a coded signal of video and outputting it to a transport stream multiplexer 27, An audio PES packetizer 24 for packetizing signals and outputting them to a transport stream multiplexer 27 and a receiver for receiving video data from the video encoder 21 and for transmitting them to the video PES packetizer 23 A buffer 25 and a reception buffer 26 for receiving audio data from the audio encoder 22 and transmitting the audio data to the audio PES packetizer 24.

In the present invention, the compensation value D _compen is obtained in the encoder device, , A time stamp compensator 28 for generating a value is introduced.

When the video encoder 21 notifies the 90-kHz counter and the time stamp compensator 28 of the time information indicating the time at which the frame image is input from the camera, the value of the 90-kHz counter is stored in the compensator 28 using this information. This value becomes E _video (0). At this time, the audio data is A / D converted (E _audio (0)). From this time, if the video AU instruction bit, which is the time information when the first byte of the video AU is input to the reception buffer 25, is input to the compensator, d _ve (0) is calculated and d _ae . d _{vbv_delay} (0) is the time required for the first image data header among the video data to accumulate by a defined amount in the virtual buffer, which may be provided with time information when the image data header is accumulated by a certain amount. In this way, we can obtain the D _compen value, which is the compensation value given in Eq. 7, and finally obtain the DTS and PTS of the compensated audio using this value. In the decoder, .

As described above, according to the present invention, the DTS of video and audio, which indicates the point in time at which the AU should be decoded, is accurately obtained and the DTS difference of video and audio is compensated in the encoder so that the synchronization between the video and the audio of the decoder can be achieved in the decoder. It is expected to be widely used in devices using MPEG-2 encoders and decoders such as HDTV, video-on-demand service, DTV, and the like.

It will be apparent to those skilled in the art that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. something to do.

Claims (3)

In a method for synchronizing MPEG-2 video and audio, A method of synchronizing video and audio using timestamp compensation, comprising compensating for audio a difference value D _compen of a decoding time stamp of video and audio of an MPEG-2 encoder, so that video and audio are synchronized in the decoder. The method according to claim 1, _Wherein the decoding time stamp difference value D _compen of the video and audio compensated for the audio is expressed by the following equation. _{D compen = d ve (0)} + d vbv_delay (0) -d ae (0) (d _ve (0): the first image input is undergoing a pre-processing the first video data is Delay before occurrence d _{vbv_delay} (0): Virtual buffer delay for the first video input d _ae (0): When the first audio is input and the first compressed audio AU data is output Time spent) A video PES packetizer for packetizing a coded signal of a video and outputting the packetized signal to a transport stream multiplexer, an audio PES packetizer for packetizing audio encoded signals and outputting the packetized audio signals to a transport stream multiplexer, An MPEG-2 encoder device comprising a receiving buffer for receiving video data from the video encoder and transmitting the video data to the video PES packetizer, and a receiving buffer for receiving audio data from the audio encoder and transmitting the data to the audio PES packetizer As a result, A video access unit indication bit indicating time information that the first byte of the video access unit is input to the reception buffer from the video encoder, an encoding start instruction bit indicating time information when a frame image is input to the encoder, A decoding time stamp difference value between audio and video is calculated by taking an audio access unit indication bit indicating the time information in which the first byte of the audio access unit is input to the reception buffer as an input and the difference value is compensated for the audio decoding time stamp And a time stamp compensator for outputting a decoding time stamp and a presentation time stamp of video and audio to the video and audio PES packetizer side.