KR20060078446A - Synchronization method for audio and video of mobile picture phone - Google Patents

Abstract

The present invention is to solve the synchronization problem of the audio and video generated by the camera in the video mode, especially in the video mode according to the video call or recording, the audio and video synchronization method of the portable video terminal If the camera is abnormally turned off during a video call through the portable video terminal, the video codec continues to perform the compression operation without stopping the video codec and counts the temporal reference value. By filling the picture layer of the stream and synchronizing audio and video, it is possible to solve the problem of inconsistency between audio and video synchronization caused by abnormal camera operation.

Video terminal, camera, TR, sync

Description

Synchronization method for audio and video of mobile picture phone}

1 is a block diagram of a mobile communication system for a mobile video call.

2 is a diagram illustrating a picture layer of an H.263 stream.

3 is a diagram illustrating an audio and video synchronization state according to a conventional video call.

4 is a diagram illustrating an audio and video synchronization error during a conventional video call.

5 is a block diagram of a portable video terminal according to an embodiment of the present invention.

6 is a diagram illustrating an audio and video synchronization state during a video call according to the present invention;

7 is a flowchart illustrating an audio and video synchronization method of a portable video terminal according to an exemplary embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

11 ... high frequency 12 ... modem chip

12a ... timer 12b ... audio codec

13 Audio processing section 14 Speaker

15 ... microphone 16 ... key input

17.DSP chip 17a ... Video codec

18 Camera part 19 LCD part

20.Memory section

The present invention relates to a method of synchronizing audio and video of a portable video terminal, in particular, to solve a problem of synchronization between audio and video generated by a camera in a video mode according to a video call or recording.

1 illustrates a typical multimedia mobile communication system. The first multimedia mobile terminal (hereinafter referred to as video terminal) 1 communicates with the second video terminal 2 via a radio link 3 to the mobile communication network 4. Control data as well as multimedia data are transmitted between the two terminals (1, 2).

The video terminals 1 and 2 encode and transmit an image captured by a camera to a counterpart for a video call. At this time, the multimedia data received from the counterpart is received, decoded by the internal codec, and played back through the display 7 and the speaker, thereby enabling a video call with the counterpart.

In conventional video telephony, the camera is not turned off while the video telephony is in use, or when the video is interrupted while sending audio video. Therefore, in this case, the audio / video synchronization is inevitably broken due to the audio / video synchronization setting method.

The synchronization setting method in the conventional video telephone is adjusted using a temporal reference (TR) value in the picture layer of H.263. The temporal criterion TR is used to set a frame rate. The interframe interval is determined according to the set frame rate, and the sum of the intervals enters the temporal basis. The temporal standard is 8 bits, and has a value between 0 and 255. When the value is larger than 255, the count starts again from zero. Since the compression of video is performed by the compression of frames one by one, each time a frame is compressed, a picture layer is generated and each time has a calculated time reference (TR) value. As a result, the video will continue to synchronize with this value until the end of video compression.

2 shows a picture layer of an H.263 stream. Each frame to be compressed has a picture layer, and after 22 bits of Picture Start Code (PSC), a time reference (TR) value is 8 bits.

The H.263 bit stream has a format as shown in FIG. The descriptor for each part is as follows.

A picture start code (PSC) indicates the start of a picture. Temporal Reference (TR) is formed by increasing the value of the previous reference picture header in time by one plus the number of omitted or non-referenced pictures since previously transmitted.

And, in addition, the PTYPE includes, among other things, the details of the picture coding type, that is, intra or inter. PQUANT is a codeword that indicates the quantizer used for the picture until updated by some subsequent quantizer information. Continued presence of CPM option A codeword that signals the use of multipoint and video multiplex (CPM) modes. PSBI picture sub-bit stream indicator-only present if CPM is set present if TRB frame is bidirectionally predicted frame present if DBQUANT bidirectional frame (known as PB-frame) exists PEI This is for extra insertion information It is set to "1" to indicate the presence of the following optional data fields PSUPP and PEI. PSUPP and PEI together are known as Supplementary Extension Information and are further defined in Annex L of H.263.

In this case, the temporal reference indicates an interval between frames according to a set bit rate or frame rate, and has a value between 0 and 255. The temporal reference value is TR (Current) = TR (Previous) + frame interval, where (0≤TR <256, TR is an integer).

Thus, the calculated temporal reference value tells us the number of input frames that need to be skipped (SKIP), which allows for synchronization between audio and video.

For example, if an H.263 compression is performed at 7.5 fps per second using a camera input at 30 frames per second (30 fps), the interval between frames would be 4 and 0, 4, 8, 12, ..., 252, 0, ...

Fig. 3 shows the synchronization between the temporal reference and audio video when compressed to 7.5 fps through a camera of 30 frames per second (30 fps) input in an ideal case, thereby enabling video telephony.

If the camera is turned off or no input is made during the video call, the video is paused in the current video call when the camera is turned off. At this time, the increase of the temporal reference value of H.263 is stopped. In this situation, the video is frozen, but the audio continues to be compressed, and the other party is listening to the voice. When the camera is turned on again, the situation as shown in FIG. 4 occurs.

As can be seen in FIG. 4, if the camera stops in a state T1 of which the temporal reference is 8, the video codec is stopped. At this time, audio is continuously outputting, and video delay (T1-T2) continues until the camera is turned on (T2). Even though the camera operates normally, the value of the temporal reference that is actually increasing does not match. Therefore, a delay (Td) occurs until the count of the temporal reference matches. At this point, the actual compressed data is combined with the audio data. This results in synchronization problems between audio and video. This is a problem that arises because the value of the temporal reference must always exist as an increasing value.

The present invention is to solve the synchronization problem caused by the on / off of the camera during the video call.

The present invention continuously compresses regardless of the camera's off to increase the temporal reference value so that the video delay does not occur due to the change of the temporal reference value, and then, at the time when the camera is turned on, the picture layer is the last temporal reference value. Is filled in, so that the synchronization between audio and video can be matched.

Audio and video synchronization method of a portable video terminal according to the present invention for achieving the above object,

In a portable video terminal,

Performing a video call;

If the camera is turned off during the video call, the video codec continues to perform the compression without stopping compression to count a temporal reference value;

And when the camera is turned on, filling the last temporal reference value of the time point into the picture layer of the video stream to synchronize audio and video.

The audio and video synchronization method of a portable video terminal according to another embodiment of the present invention increases the count of a temporal reference value by continuously compressing a video codec regardless of an abnormal error of a camera in a video mode of the terminal. step;

If the camera operates normally, the video stream is filled with a temporal reference value, which is counted at the last time, to synchronize audio and video.

Preferably, the video stream is characterized in that the H.263 stream.

4 is a block diagram illustrating an audio and video synchronization device of a portable video terminal according to an exemplary embodiment of the present invention.

Referring to Fig. 4, the radio frequency unit (RF unit) 11 performs a wireless communication function of the cellular phone. The high frequency unit 11 includes an RF transmitter for upconverting and amplifying a frequency of a transmitted signal, and an RF receiver for low noise amplifying and downconverting a received signal.

The modem chip 12 includes a transmitter for encoding and modulating the transmitted signal, a receiver for demodulating and decoding the received signal, and the like. That is, the modem chip 12 includes a modem and an audio codec. The codec includes a data codec for processing packet data and the like and an audio codec 12b for processing an audio signal such as voice. In addition, the audio codec 12b is a speech codec embedded in the mobile phone, and encodes an input voice signal during a call with the other party. The MSM chip 12 controls the video call mode as a whole and provides time information of a clock generated by the timer 12a.

The audio processor 13 reproduces the received audio signal output from the audio codec 12b of the modem chip 12 to the speaker 13 or the input audio signal generated from the microphone 15 to the modem chip 12. Performs the function of transmitting to the audio codec 12b.

The key input unit 16 includes keys for inputting numeric and character information and function keys for setting various functions. In addition, the key input unit 16 may include function keys for controlling a video recording mode and photographing keys for driving a camera according to an exemplary embodiment of the present invention. In addition, in the exemplary embodiment of the present invention, the text signal synthesized to the composite video signal may be input through the key input unit 16.

The digital signal processor (DSP) chip 17 encodes screen data for displaying an image signal captured by the camera unit 18 to be transmitted to the other party. The DSP chip 17 processes the image signal output from the camera unit 18 in units of frames, and outputs the frame image data according to the characteristics and size of the LCD unit 19. In addition, the DSP chip 17 includes a video codec 17a. The video codec 17a is, for example, H.263, and encodes an image signal in a set manner or encodes the encoded frame image data. Decodes the frame image data. The DSP chip 17 generates and reproduces a video signal according to a mode set under the control of the MSM chip 12.

The camera module 18 includes a camera sensor for converting an optical signal sensed during image capturing into an electrical signal, and a signal processor for converting an analog image signal captured from the camera sensor into digital data. Herein, the camera sensor is assumed to be a CCD sensor, and the camera sensor and the signal processor may be embodied integrally or separately.

The LCD unit 19 displays a frame image signal or a counterpart image output from the DSP chip 17 on a screen, and displays user data output from the MSM chip 12. The LCD unit 19 also displays a video signal reproduced under the control of the DSP chip 17.

The memory unit 20 may include a program memory and a data memory. The program memory may store programs for controlling a general operation of the mobile phone and programs for controlling a path of an image signal applied to the LCD unit. In addition, the data memory temporarily stores data generated during the execution of the programs. The data memory may include a text buffer for storing text signals input through the key input unit 16, an image buffer for storing a video signal generated by the video codec 17a, and an audio signal generated by the audio codec 12b. It may be provided with audio buffers for storing the. The video memory is a memory for storing composite data of image data and text signals (which may include audio signals) coded according to an exemplary embodiment of the present invention.

In the present invention, even if the camera is turned off during the video call, the video codec 17a continues to perform compression regardless of the camera being turned off. Accordingly, compression continues even after the camera is turned off, so that the temporal reference value is constantly increased over time, and then, when the camera is turned on, the picture layer is filled with the last temporal reference value at the time the camera is turned on so that the video is turned off. Even if it is delayed, it can solve the problem of losing the synchronization of audio and video.

Referring to FIG. 6, when the time T11 at which the camera is turned off is 8 for the temporal reference value, the video codec continues to compress while the video call is in progress. As the video is compressed, the temporal reference value is passed on.

Subsequently, if the last temporal reference value is filled in at the time T12 when the camera is turned on, it is possible to solve the problem that the synchronization between the audio and the video is broken even when the camera is turned off and a video delay occurs. For example, synchronization is achieved by filling the picture layer with the last temporal reference value at the time T12 when the camera is turned on.

7 is a flowchart illustrating a video and audio synchronization method of the portable video terminal of the present invention.

Referring to FIG. 7, a video call is performed (S11). At this time, it is determined whether the camera is turned off (S13), and when the camera is turned off, the video codec continues to compress and increases the TR value (S15). After that, it is determined whether the camera is turned on again (S17), and when the camera is turned on, the last TR value is filled in the picture layer (S19) to synchronize video and audio.

The present invention allows the video codec to continue to operate even if the camera is operating abnormally in the recording mode as well as the video telephone, so that the temporal reference value is continuously increased, so that audio and video can be synchronized with the normal operation. have.

So far, the present invention has been described with reference to the preferred embodiments, and those skilled in the art to which the present invention pertains to the detailed description of the present invention and other forms of embodiments within the essential technical scope of the present invention. Could be implemented. Here, the essential technical scope of the present invention is shown in the claims, and all differences within the equivalent range will be construed as being included in the present invention.

As described above, according to the method of synchronizing audio and video of a portable video terminal according to the present invention, the audio codec may be generated between video and audio by continuously operating the video codec regardless of whether the camera is on or off during a video call. It is effective to reduce the motivation.

In addition, if the audio and video synchronization is broken due to the temporary stop of the camera, the audio screen comes first during the video call, and then the video screen appears, thereby preventing the eyes and ears from being disturbed.

Claims (4)

In a portable video terminal, Performing a video call; If the camera is turned off during the video call, the video codec continues to perform the compression without stopping compression to count a temporal reference value; And when the camera is turned on, filling the last temporal reference value of the time point into the picture layer of the video stream to synchronize audio and video. The method of claim 1, The delay of the video is delayed only from the time when the camera off to the time when the camera on audio and video synchronization method of the portable video terminal. In a portable video terminal, Increasing the count of the temporal reference value by continuously compressing the video codec regardless of an abnormal error of the camera in the video mode of the terminal; And synchronizing audio and video by filling a video stream with a temporal reference value, which is last counted at that time, when the camera is normally operated. 2. The method according to claim 1 or 3, And the video stream is an H.263 stream.

Images