KR20060060109A - Synchronization apparatus and method for audio and video of portable terminal - Google Patents

Abstract

The present invention relates to a mobile terminal that supports VOD (Video On Demand), in particular to enable synchronization by forcibly setting the bit information in the received data frame, for audio and video synchronization received in real time, The apparatus for synchronizing audio and video of a portable terminal according to the present invention checks synchronization of received multimedia packet data by audio and video time stamps and sets predetermined information in the data frame when the time difference is greater than a threshold. And a signal processing chip which performs decoding from the next frame by jumping the corresponding frame according to the information set in the data frame.

Handset, audio, video, sync, NAL, FZB

Description

Synchronization apparatus and method for audio and video of portable terminal

1 is a block diagram of a conventional portable terminal.

2 is a block diagram showing an audio and video synchronization device of a mobile terminal according to an embodiment of the present invention.

3 is a detailed block diagram of a signal processing chip according to an embodiment of the present invention.

4 illustrates NAL unit syntax in accordance with the present invention.

5 is a flowchart illustrating an audio and video synchronization method of a mobile terminal according to the present invention;

6 is a flowchart showing an error processing routine by the signal processing chip in the mobile terminal according to the present invention.

<Description of the symbols for the main parts of the drawings>

100 ... handset 110 ... modem chip

111 Protocol processing module 112 Interface

117 ... RTP processing unit 118 ... Packet restoration module

120 ... Signal Processing Chip 121 ... Synchronous Check Module

121a ... Frame buffer 122 ... File system storage

123 ... Codec 124 ... Playback

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for audio and video synchronization, in particular, which is received in real time in a portable terminal supporting Video On Demand (VOD).

In general, video signal encoding techniques are used for the transmission, storage, and playback of video signals. For example, international standard video decoding such as ITU-T Recommendation H.263 (hereinafter referred to as 'H.263'), ISO / IEC International Standard 14496-2 (MPEG-4 Visual, hereinafter referred to as 'MPEG-4'), etc. Methods. And new encoding systems include ITU-T and ISO / IEC. That is, video designed for joint international standardization by the ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10 (Joint Final Committee Draft, H.26L). Coding methods, including general coding techniques used in these video coding methods.

Due to the development of the compression technology of the video signal and the recent development of the technology of the portable terminal, a video on demand (VOD) function is provided to the terminal. On-demand video function is not a function of downloading and playing back VOD contents in VOD server on the Internet, but receiving and processing streaming data sent from VOD server in real time and showing contents to user. . Here, the VOD content refers to content transmitted to the terminal through the VOD function among contents in the VOD server.

In addition, on-demand video capabilities enable movie trailers, music videos, and news, as well as television broadcasts in recent years. In addition, as the storage medium capacity of the terminal increases, the terminal can simultaneously record and record VOD contents.

1 is a block diagram of a conventional H.26L-based VOD terminal.

Referring to FIG. 1, the mobile terminal 10 generally includes a mobile station modem (MSM) chip 20 and a signal processing chip 30, and the modem chip 10 encodes and modulates a transmitted signal. A transmitter, a receiver for demodulating and decoding the received signal, and a protocol processing module 21. In this case, the protocol processing module 21 processes the received multimedia packet data.

Here, the multimedia packet (RTP packet) data includes header to point information such as Point to Point Protocol (PPP), Internet Protocol (IP), Transport Control Protocol / User Datagram Protocol (TCP / UDP), Real Time Protocol (RTP) / RTSP (Real). Time Streaming Protocol) / HTTP is inserted, and multimedia data (H.264) is carried in the RTP payload.

Accordingly, when the RTP packet is received, the protocol processing module 21 processes the PPP, IP, and TCP / UDP protocols of the lower layer, and transmits multimedia data corresponding to the RTP packet to the signal processing chip 30. Will be delivered.

In addition, the modem chip 10 may include a codec for decoding MIDI and audio data.

The signal processing chip 30 includes a file system 31, a codec 32, and a player 33. The file system 31 stores multimedia packet data, and the codec 32 is a video. Decode the multimedia packet data for playback, and player 33 controls the display of the decoded multimedia data for real-time video service.

Here, the interface method of the modem chip 20 and the signal processing chip 30 is connected to, for example, an IPC (Inter Process Communication) 22.

The signal processing chip 30 may process all H.264 based bit streams. That is, on-demand video or real-time video service adopts H.264 as a video codec 32.

The signal processing chip 30 decodes all of MP3, AAC (Advanced Audio Coder), etc., and speech codec (QCELP, EVRC) corresponding to an audio codec other than the MIDI level.

Such a mobile terminal has higher priority of audio than video when audio / video is simultaneously played, so that the sound is heard to the user even if the video is interrupted. That is, when video and audio data coexist simultaneously, audio is preferentially processed and video can be occupied after the audio data is processed. If the audio frequency is high, the data has to be processed frequently, so video has less opportunity to occupy the CPU, that is, the signal processing chip.

Accordingly, in the conventional portable terminal, when audio and video are played simultaneously, the audio data has a relatively large share of the signal processing chip 30 due to the high bit rate of the audio, so that if the video decodes all the bit streams, the decoding is performed. Depending on the time spent, the time synchronization between the audio and the video does not match.

H.264 also has more complexity than MPEG 4. In other words, H.264 performs the entire search within 32 (forward 16 and backward 16) by combining before and after the most similar to the current macroblock. I have it. On the other hand, MPEG 4 (MPEG 4) shows that H.264 has high complexity when compared to processing two sheets before and after for B-picture. Due to the complexity of H.264 and the priority of audio, the signal processing chip is heavily loaded, and such a load is more deadly in real time broadcasting.

Therefore, in the related art, there is a problem that a time difference occurs between audio and video generated when the audio signal and the video signal are reproduced at the same time due to the bit rate of the audio, which is relatively small, which can occupy the video signal processing chip.

 A first object of the present invention is to provide an audio / video synchronization apparatus and method for H.264 based real-time video service.

A second object of the present invention is to enable audio / video synchronization without additional memory or processor overhead associated with audio / video synchronization.

A third object of the present invention is to enable the synchronization check module to forcibly set bits in the NAL schedule information so as to synchronize the audio and video when the time difference between the audio and the video is fixed.

A fourth object of the present invention is to perform an error processing or cancellation routine using the set bit information in a codec, thereby jumping frame data into which the bit information is inserted into the next frame data without performing a decoding process, thereby performing audio and To keep the video in sync.

Audio and video synchronization device of a mobile terminal according to the present invention for achieving the above object

A receiving unit for receiving multimedia packet data;

A synchronization check module for checking synchronization by using a time difference by an audio and video time stamp according to reproduction of the received multimedia data, and setting predetermined information in the data frame when the time difference is greater than or equal to a threshold;

A codec for decoding for reproduction of the data frame, and performing decoding from a next frame by jumping a current frame according to information set in the data frame during decoding;

And a reproduction unit for reproducing the data decoded by the codec.

Preferably, the synchronization check module is characterized by using the current audio and video time stamp transmitted from the playback unit.

In detail, the synchronization check module sets predetermined bit information to FZB (Forbidden Zero Bit) information of a network abstraction layer (NAL) when a time difference between audio and video is greater than or equal to a threshold.

Preferably, the synchronization check module is characterized by setting predetermined information in the next data frame when the time difference between audio and video is more than a certain threshold less than the period of the NAL frame.

Preferably, the codec is characterized in that the H.264.

An audio and video synchronization device of a mobile terminal according to another embodiment of the present invention,

A modem chip for receiving multimedia packet data; Means for setting synchronization bit information in an arbitrary data frame for synchronization of audio and video data according to reproduction of the received packet data, and after checking the synchronization bit information when decoding the data frame And a signal processing chip including a means for jumping from a data frame set with bit information and performing decoding from the next frame.

The audio and video synchronization method of the mobile terminal according to the present invention,

Receiving multimedia packet data; Detecting an audio and video time stamp according to reproduction of the received packet data; Confirming whether a difference between the audio and video time stamps is greater than or equal to a threshold, and if the verification result is greater than or equal to a threshold, setting bit information in FZB information of a NAL frame;

Checking the FZB information of the NAL frame when decoding the packet data, jumping the corresponding data frame according to the bit information set in the FZB information of the NAL frame, and performing decoding from the next data frame. .

Hereinafter, with reference to the accompanying drawings as follows.

2 is a block diagram illustrating an audio and video synchronization device of a mobile terminal according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the portable terminal 100 forcibly sets synchronization bit information in an arbitrary frame for synchronization of audio and video according to reproduction of data with the modem chip 110 receiving packet data. The signal processing chip 120 is configured to synchronize audio and video by processing (jumping or passing) the corresponding frame into which the information is inserted.

The modem chip 110 includes a protocol processing module 111. The signal processing chip 120 includes a synchronization check module 121, a file system storage unit 122, a codec 123, and a playback unit 124. It will be included.

The sync check module 121 checks the audio and video time stamps and notifies the codec 123 when the difference is out of the threshold, and the codec 123 has a high bit stream input. Determining the complexity, the current frame is jumped and decoding is performed from the next frame.

Here, as an example of notifying the codec 123, the synchronization check module 121 sets bit information to "jump" because the frame is not required in the FZB information of the NAL unit forcibly. The NAL FZB information is continuously checked to jump the current frame according to the set bit information and to decode the next frame.

Referring to the accompanying drawings, an audio and video synchronization device of a mobile terminal according to an embodiment of the present invention configured as described above will be described.

As shown in FIG. 2, when a multimedia packet, that is, a Real Time Protocol (RTP) packet, is received by the mobile terminal 100, the modem chip 120 processes various protocols by an internal protocol processing module 121. Done. Here, the RTP packet is received with PPP, IP, and TCP / UDP protocols as a lower layer in the header information, and includes RTP / RTSP / HTTP as an actual header, and multimedia data is carried in the payload.

The multimedia data processed by the modem chip 110 is transmitted to the signal processing chip 120 through the interface 122, and the signal processing chip 120 decodes and stores the multimedia data, and reproduces the multimedia data. Synchronize audio and video data.

To this end, the signal processing chip 120 includes a synchronization check module 121, a file system storage unit 122, a codec 123, and a playback unit 124, and the synchronization check module 121 includes audio and video. The time stamp is used to detect a time difference between audio and video. When the time difference between the audio and video is greater than or equal to a threshold, bit information is forcibly set in NAL (hereinafter, referred to as "day") information.

In addition, the synchronization check module 121 may continuously check synchronization, or check synchronization at regular intervals. If the time stamps of audio and video deviate from the threshold, the synchronization check module 121 may select whether to jump or delete the packet data frame. Will be set. Alternatively, when bit information is set in the current frame, it may be instructed to actually jump to the next data frame.

Here, the NAL unit syntax of H.26L (eg, H.264) is as shown in FIG. 4. In the present embodiment, it is defined in a day unit syntax that carries information on a day (NAL), which is a unit for transmitting H.26L encoded data. Specifically, this is defined as the type known by the raw unit type included in the raw unit syntax.

As shown in FIG. 4, one byte of the day unit syntax 140 indicates whether each day type (NAL_unit_type) 143 and the corresponding day reference (NAL_ref_idc) 142 are used. Here, the FZB (Forbidden_Zero_Bit) information 141 corresponding to the first byte is fixed to '0 (zero)' in ISO 14496-10 and is not used.

However, in the packet network, since the FZB information 141 corresponding to bit error free is a bit meaningless, the present invention effectively stores the FZB information 141 of the NAL unit in the received RTP packet. I want to use That is, the FZB information 141 of the raw unit can be used to effectively instruct the codec 123 of an error (for example, packet loss, etc.) at the channel end.

The codec 123 is H.26L, which decodes multimedia data passing through the synchronization check module 121, and the decoded data is reproduced by the reproducing unit 124. In this case, the audio and video time stamps reproduced by the playback unit 124 are fed back to the synchronization check module 121.

At this time, the codec 123 checks the FZB information of the day unit of the multimedia data passing through the synchronization check module 121. If the FZB of the raw unit is set ("1") so that the NAL unit is configured, the codec 123 recognizes that at least one packet loss occurred before the current NAL unit arrives. The IDR NAL unit frame is jumped without decoding, and the next IDR NAL unit frame is decoded. In other words, audio and video synchronization can be achieved by jumping the frame or passing the frame.

That is, the signal processing chip 120 includes an error handling routine, and the codec 123 decodes when information previously promised is set in the FZB information of the raw unit by the error handling routine. In principle, freezing is done until the next possible NAL, i.e., the IDR NAL, arrives. For example, in H.26L, the next frame that can be decoded, i.e., the current IDR picture frame in which the raw unit's FZB is set until the instantaneous decoder refresh (IDR) picture arrives or until it is decoded. I jump without doing it.

In addition, the file system storage unit 122 stores the multimedia data passing through the synchronization check module 122, and when it is to be reproduced again, the file system storage unit 122 is decoded by the codec 123 and reproduced by the reproduction unit 124. .

3 is a detailed block diagram of a signal processing chip according to an exemplary embodiment of the present invention.

Referring to FIG. 3, when the RTP packet is delivered to the RTP processing unit 117, the RTP processing unit 117 separates the RTP header, and the packet reconstruction module 118 restores the packet's payload and header. do. The synchronization check module 121 receives the audio and video time stamps and calculates a time difference. When the calculated time difference is less than a threshold, the synchronization check module 121 passes the FZB information of the NAL unit without setting.

However, if the time difference between the audio and video time stamp is more than the threshold value, the FZB information of the NAL unit is forcibly set. Here, the threshold value is defined as, for example, 1000 ms. This is because the VOD content standard inserts an IDR NAL frame at least once every 2 seconds, and thus jumps by the forcibly inserted FZB information. Accordingly, the codec 213 can earn a decoding time equal to or smaller than the IDR NAL frame insertion period (eg, 1000 ms), so that the next frame can be smoothly reproduced.

The frame buffer 121a stores the NAL frame reconfigured by the packet recovery module 118. Here, one NAL frame consists of one or more RTP payloads.

In order to decode the raw frame, the codec 123 requests the next day frame to the frame buffer 121a and decodes the requested day frame. At this time, when decoding the raw frame, the FZB information of the raw unit is checked to determine whether to decode or jump. For example, if raw FZB information is set, it is determined that the bit stream input to the codec 123 has a high complexity, and continuous decoding thereof makes it impossible to synchronize audio / video. Jumping to the frame will decode the next IDR NAL frame.

Accordingly, since there is no use of memory for notifying a channel loss (packet loss) and the error handling routine used basically is used as it is, a separate module is not required.

5 is a flowchart illustrating an audio and video synchronization method according to the present invention.

Referring to FIG. 5, when the RTP packet is received by the terminal and the received frame is reproduced (S101), time stamps of audio and video are checked (S103).

At this time, it is checked whether or not the audio and video time stamp difference (= audio time stamp-video time stamp) is greater than or equal to the threshold value (S105). If the verification result is less than the threshold value, the decoding and playback operations are continuously performed.

If the time stamp difference is greater than or equal to the threshold value, information for synchronization is set in the FZB information of the NAL unit (S107). Here, the information for synchronization may simply jump or delete the corresponding NAL frame.

Then, the codec performs decoding (S109). At this time, when decoding, it is checked whether the FZB information of the raw unit is set (S111). If the FZB information of the raw unit is set as a result of the check, it jumps to the next IDR NAL frame and decodes from the frame (S113). .

6 is a general flow diagram for audio and video synchronization in accordance with the present invention.

Referring to FIG. 6, the frame of the frame buffer is patched in the initialization state S121 (S123), and the raw unit is analyzed (S131). In step S131, the blade unit is analyzed to determine whether the FZB information of the blade unit is set (S131). If the FZB information is set as a result of the check, the next IDR NAL frame is jumped to (S133). If the FZB information is not set or jumped to the next IDR NAL frame, the header is parsed (S135), and the data (Motion information, Residual information) is decoded and reproduced (S137).

At this time, the synchronization check module checks the synchronization of the audio and video time according to the playback of the decoded frame and the time information from the frame buffer, thereby checking the difference in delay time between the audio and the video (S125).

As a result of the check in S125, it is determined whether the time when the video time is delayed than the audio time is greater than or equal to the threshold (S127). If less, the step S123 is performed without setting the FZB information of the raw unit (S129).

Through this error handling routine, it is indicated whether or not a raw frame is jumped (or passed) using FZB information of the raw unit so that time / synchronization of audio / video can be matched. Pass), the audio and video signals are synchronized during reproduction.

In other words, by forcibly setting sync bit information in an arbitrary data frame and processing the same in a codec, the video can follow the audio.

So far, the present invention has been described with reference to the preferred embodiments, and those skilled in the art to which the present invention pertains to the detailed description of the present invention and other forms of embodiments within the essential technical scope of the present invention. Could be implemented. Here, the essential technical scope of the present invention is shown in the claims, and all differences within the equivalent range will be construed as being included in the present invention.

As described above, according to the audio and video synchronization device and method of the mobile terminal according to the present invention, if the time difference between the audio and video is more than a predetermined time difference, by setting a predetermined command to a specific information bit of the raw unit, the codec When decoding the above information, it is possible to jump from the current frame and decode the next frame according to the corresponding command, so that audio and video synchronization can be achieved.

In addition, there is an effect that enables smooth playback of audio and video through the codec without adding a separate memory or process according to the synchronization of audio and video.

Claims (7)

A receiving unit for receiving multimedia packet data; A synchronization check module for checking synchronization using a time difference based on an audio and video time stamp according to reproduction of the received packet data, and setting predetermined information in the data frame when the time difference is greater than or equal to a threshold; A codec for decoding for reproduction of the data frame, and performing decoding from a next frame by jumping a current frame according to information set in the data frame during decoding; And a reproducing unit for reproducing the data decoded by the codec. The method of claim 1, And the synchronization check module uses a current audio and video time stamp transmitted from the playback unit. The method of claim 1, And the synchronization check module sets predetermined bit information to FZB (Forbidden Zero Bit) information of a network abstraction layer (NAL) when a time difference between audio and video is greater than or equal to a threshold.  The method of claim 1, And the synchronization check module sets predetermined information in the next data frame when the time difference between the audio and the video is greater than or equal to a certain threshold less than the NAL frame insertion period. The method of claim 1, And the codec is H.264. A modem chip for receiving multimedia packet data; Means for setting synchronization bit information in an arbitrary data frame to synchronize audio and video data according to reproduction of the received packet data, and for synchronization after checking the synchronization bit information when decoding the data frame. And a signal processing chip including a means for jumping from a data frame set with bit information to perform decoding from the next frame. Receiving multimedia packet data; Detecting an audio and video time stamp according to reproduction of the received packet data; Confirming whether a difference between the time stamps of the audio and video is greater than or equal to a threshold, and if the verification result is greater than or equal to a threshold, setting information in the FZB information of the NAL frame; And checking the FZB information of the NAL frame when decoding the packet data, jumping the corresponding data frame according to the information set in the FZB information of the NAL frame, and performing decoding from the next data frame. How to synchronize audio and video on your device.

Images