CN108243350B

CN108243350B - Audio and video synchronization processing method and device

Info

Publication number: CN108243350B
Application number: CN201611225383.8A
Authority: CN
Inventors: 郭荣
Original assignee: Sanechips Technology Co Ltd
Current assignee: Sanechips Technology Co Ltd
Priority date: 2016-12-26
Filing date: 2016-12-26
Publication date: 2020-05-05
Anticipated expiration: 2036-12-26
Also published as: WO2018120557A1; CN108243350A

Abstract

The embodiment of the invention discloses a method for audio and video synchronous processing, which comprises the following steps: comparing the current program clock reference PCR value with the current system clock STC value to obtain a first comparison result; when the first comparison result does not meet the preset judgment condition, reconstructing a system clock STC by adjusting the current STC counting step length; comparing the absolute value of the difference between the STC basic value STC _ base in the reconstructed system clock STC and the display time stamp PTS value with a preset time interval to obtain a second comparison result; the preset time interval is M times of the inter-frame time of the audio or video, and M is greater than 0 and smaller than 1; and adjusting the audio or video playing speed according to the second comparison result. The embodiment of the invention also discloses a device for synchronously processing the audio and the video.

Description

Audio and video synchronization processing method and device

Technical Field

The invention relates to the field of digital televisions, in particular to a method and a device for audio and video synchronous processing.

Background

The encoder has a System Time Clock (STC) with a frequency of 27MHz, and the STC is used to generate a PTS (Presentation Time Stamp) and a DTS (Decode Time Stamp) for audio and video, and also used to indicate an instantaneous sampling value of the STC. The transmitting end inserts the instantaneous sampling value of the system Clock into the PCR domain of the MPEG-2 TS stream at the moment that the Program Clock Reference field leaves the multiplexer, the receiving end can recover the 27MHz system Clock synchronous with the encoder by extracting the PCR field, and then realizes the synchronization of the audio and the video by using the PTS and the DTS in the PES stream.

The PCR value in the MPEG-TS stream is 42 bits, including 33 bits of PCR _ base and 9 bits of PCR _ extension. The PCR _ base is sampled by a 90KHz clock, and the function of the PCR _ base is to provide an initial value of local STC counting when switching programs, so that the PCR value and PTS and DTS have the same time starting point as much as possible. PCR _ extension is sampled with a 27MHz clock, and functions to correct the system clock of the decoder through a phase-locked loop at the receiving end, thereby obtaining a 27MHz system clock consistent with the encoder.

However, in a real-time audio/video playing system, lip synchronization sometimes occurs, mainly due to inaccuracy of the reconstructed system clock.

Disclosure of Invention

In order to solve the above technical problems, embodiments of the present invention desirably provide a method and an apparatus for audio and video synchronization processing, which can accurately reconstruct a system clock consistent with a coding end at a decoding end, so as to provide an accurate clock reference for audio and video synchronization, and then solve the problem of lip-sound asynchrony in a video playing process through the audio and video synchronization processing.

The technical scheme of the invention is realized as follows:

in a first aspect, an embodiment of the present invention provides an audio and video synchronization processing method, where the method includes:

comparing the current program clock reference PCR value with the current system clock STC value to obtain a first comparison result;

when the first comparison result does not meet the preset judgment condition, reconstructing a system clock STC by adjusting the current STC counting step length;

comparing the absolute value of the difference between the STC basic value STC _ base in the reconstructed system clock STC and the display time stamp PTS value with a preset time interval to obtain a second comparison result; the preset time interval is M times of the inter-frame time of the audio or video, and M is greater than 0 and smaller than 1;

and adjusting the audio or video playing speed according to the second comparison result.

In the foregoing solution, the comparing the current program clock reference PCR value with the current system clock STC value to obtain a first comparison result includes:

loading the first PCR value or the reloaded PCR value as a counting initial value into a local STC counter;

driving the local STC counter by using the local system clock frequency to count STC in an increasing way; the local system clock frequency is N times of the coding system clock frequency, and N is an integer greater than or equal to 1;

reading a current STC value and the current PCR value in the local STC counter when PCR interruption is detected;

calculating the absolute value of the difference between the current PCR value and the current STC value in the local STC counter; wherein an absolute value of a difference between the current PCR value and the current STC value in the local STC counter is the first comparison result.

In the foregoing solution, when the first comparison result does not satisfy the preset determination condition, reconstructing the system clock STC by adjusting the current STC count step size includes:

comparing the first comparison result with a preset judgment condition;

when the first comparison result does not meet a preset judgment condition, comparing the current PCR value with the current STC value;

if the current PCR value is larger than the current STC value, increasing the current STC counting step length;

and if the current PCR value is smaller than the current STC value, reducing the current STC counting step length.

In the above aspect, the method further includes:

and when the absolute value of the difference value between the current PCR value and the current STC value meets a preset judgment condition, keeping the STC counting step length unchanged.

In the above solution, when decoding a video, the adjusting the video playing speed according to the second comparison result includes:

correspondingly adjusting the decoding speed of the video by using the second comparison result;

when the absolute value of the difference value between the STC _ base value and the PTS value is smaller than the preset time interval, the video decoding speed is not adjusted;

and when the absolute value of the difference value between the STC _ base value and the PTS value is greater than or equal to the preset time interval, adjusting the video decoding speed.

In the above scheme, when the absolute value of the difference between the STC _ base value and the PTS value is greater than or equal to the preset time interval, adjusting the video decoding speed includes:

when the difference value obtained by subtracting the PTS value from the STC _ base value is larger than or equal to the preset time interval, the video decoding speed is smaller than the preset normal speed, and corresponding synchronous adjustment is carried out according to the structure of the video frame; wherein, the structure of the video frame is respectively as follows: bidirectional predicted frame B frame, intra-frame predicted frame I frame and unidirectional predicted frame P frame;

and when the difference value obtained by subtracting the STC _ base value from the PTS value is larger than or equal to the preset time interval, the video decoding speed is larger than the preset normal speed, the last video frame is repeatedly displayed, the video decoding is suspended, and the current image is displayed when the STC _ base value is the same as the PTS value.

In the above solution, when a difference obtained by subtracting the PTS value from the STC _ base value is greater than or equal to a preset time interval, the video decoding speed is less than a preset normal speed, and performing corresponding synchronization adjustment according to the structure of the video frame includes:

when the video frame is a B frame, discarding the current frame, decoding the next frame, and displaying the previous frame at the same time;

and when the video frame is an I frame or a P frame, freezing the image of the previous frame, stopping decoding, and waiting for the next I frame or video sequence.

In the above solution, when decoding an audio, the adjusting the audio playing speed according to the second comparison result includes:

correspondingly adjusting the decoding speed of the audio by using the second comparison result;

when the absolute value of the difference value between the STC _ base value and the PTS value is smaller than the preset time interval, the audio decoding speed is not adjusted;

and when the absolute value of the difference value between the STC _ base value and the PTS value is greater than or equal to the preset time interval, adjusting the audio decoding speed.

In the above scheme, when the absolute value of the difference between the STC _ base value and the PTS value is greater than or equal to the preset time interval, adjusting the audio decoding speed includes:

when the difference value obtained by subtracting the PTS value from the STC _ base value is larger than or equal to the preset time interval, the audio decoding speed is smaller than a preset normal speed, and one sampling point is skipped;

and when the difference value obtained by subtracting the STC _ base value from the PTS value is larger than or equal to the preset time interval, the audio decoding speed is larger than a preset normal speed, and the last sampling point is repeated.

In a second aspect, an embodiment of the present invention provides an apparatus for audio and video synchronization processing, where the apparatus includes: a comparison module and an adjustment module; wherein,

the comparison module is used for comparing the current program clock reference PCR value with the current system clock STC value to obtain a first comparison result;

the adjusting module is used for reconstructing a system clock STC by adjusting the current STC counting step length when the first comparison result does not meet the preset judgment condition;

the comparison module is further configured to compare an absolute value of a difference between an STC basis value STC _ base and a display time stamp PTS value in the reconstructed system clock STC with a preset time interval to obtain a second comparison result; the preset time interval is M times of the inter-frame time of the audio or video, and M is greater than 0 and smaller than 1;

the adjusting module is further configured to adjust the audio or video playing speed according to the second comparison result.

In the above solution, the apparatus further comprises: the device comprises a loading module, a counting module, a reading module and a calculating module; wherein,

the loading module is used for loading the first PCR value or the reloaded PCR value as a counting initial value into a local STC counter;

the counting module is used for driving the local STC counter by using the local system clock frequency to count STC in an increasing way; the local system clock frequency is N times of the coding system clock frequency, and N is an integer greater than or equal to 1;

the reading module is used for reading the current STC value and the current PCR value in the local STC counter when PCR interruption is detected;

the calculation module is used for calculating the absolute value of the difference between the current PCR value and the current STC value in the local STC counter; wherein an absolute value of a difference between the current PCR value and the current STC value in the local STC counter is the first comparison result.

In the above scheme, the comparing module is configured to compare the first comparison result with a preset judgment condition;

the adjusting module is used for comparing the current PCR value with the current STC value when the first comparison result does not meet a preset judgment condition;

and if the current PCR value is larger than the current STC value, increasing the current STC counting step length;

In the above solution, the apparatus further comprises: a holding module; wherein,

and the keeping module is used for keeping the STC counting step length unchanged when the absolute value of the difference value between the current PCR value and the current STC value meets a preset judgment condition.

In the above solution, the adjusting module is configured to perform corresponding adjustment on the decoding speed of the video by using the second comparison result;

and when the absolute value of the difference value between the STC _ base value and the PTS value is smaller than the preset time interval, the video decoding speed is not adjusted;

In the foregoing solution, the adjusting module is configured to, when a difference obtained by subtracting the PTS value from the STC _ base value is greater than or equal to the preset time interval, perform corresponding synchronization adjustment according to a structure of the video frame, where the video decoding speed is less than a preset normal speed; wherein, the structure of the video frame is respectively as follows: bidirectional predicted frame B frame, intra-frame predicted frame I frame and unidirectional predicted frame P frame;

In the above solution, the adjusting module is configured to discard the current frame, decode the next frame, and display the previous frame when the video frame is a B frame;

In the above solution, the adjusting module is configured to perform corresponding adjustment on the decoding speed of the audio by using the second comparison result;

and when the absolute value of the difference value between the STC _ base value and the PTS value is smaller than the preset time interval, the audio decoding speed is not adjusted;

In the above solution, the adjusting module is configured to skip a sampling point when a difference obtained by subtracting the PTS value from the STC _ base value is greater than or equal to the preset time interval, and the audio decoding speed is less than a preset normal speed;

The embodiment of the invention provides a method and a device for audio and video synchronous processing, when the comparison result of a current PCR value and a current STC value does not meet a preset judgment condition, the current STC counting step length is adjusted to reconstruct a system clock STC, then the audio/video playing speed is adjusted according to the comparison result of the absolute value of the difference value between the STC _ base value in the reconstructed system clock STC and the PTS value of a display time label and a preset time interval, and a system clock consistent with an encoding end is accurately reconstructed at a decoding end, so that accurate clock reference is provided for audio and video synchronization, and the problem of lip-sound asynchronization in the video playing process is solved.

Drawings

Fig. 1 is a schematic diagram of an apparatus for reconstructing a local system clock and performing audio/video synchronization processing according to an embodiment of the present invention;

fig. 2 is a first schematic flow chart of an audio and video synchronization processing method according to a first embodiment of the present invention;

fig. 3 is a schematic flow chart of an audio and video synchronization processing method according to the first embodiment of the present invention;

fig. 4 is a schematic flow chart of an audio and video synchronization processing method provided in the first embodiment of the present invention;

fig. 5 is a flowchart of system clock reconstruction using a PCR and a local STC counter according to an embodiment of the present invention;

fig. 6 is a schematic flow chart of an audio and video synchronization processing method according to a first embodiment of the present invention;

fig. 7 is a schematic flow chart of an audio and video synchronization processing method according to a first embodiment of the present invention;

fig. 8 is a flowchart of audio and video synchronization using STC _ base and PTS according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a first audio/video synchronization processing apparatus according to a second embodiment of the present invention;

fig. 10 is a schematic structural diagram of an audio/video synchronization processing apparatus according to a second embodiment of the present invention;

fig. 11 is a hardware block diagram of audio and video synchronization processing provided in the third embodiment of the present invention.

Detailed Description

The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

The idea of the invention is as follows: in a multimedia system working in real time, the problem of error in audio and video synchronization caused by inaccurate reconstruction of a system clock is solved. And by reconstructing the local system clock, audio and video synchronization processing is carried out on the basis of reconstructing the local system clock.

Referring to fig. 1, a schematic diagram of an apparatus for reconstructing a local system clock and performing audio/video synchronization processing according to an embodiment of the present invention is shown. The device needs to establish an STC counter locally, when a first PCR arrives or the PCR needs to be reloaded, the current PCR value is loaded into the STC counter, the STC counter counts under the drive of a 108MHz clock, after the interruption of the PCR is detected, the value of the STC counter is read and compared with the current PCR value, if the deviation is within an acceptable range, no adjustment is carried out, otherwise, software needs to adjust the STC counting step length to adjust the STC counter, and further, the system clock is corrected. And a low-pass filter and a voltage-controlled oscillator are used for phase locking, so that large deviation caused by transmission error codes is avoided.

The clock driving the local STC counter in the device is selected to be 108MHz, and the encoding clock is 27MHz, which is 4 times the encoding end clock. The frequency is increased to improve the precision of the counter, and meanwhile, the problem that the reconstructed system clock is not accurate due to inaccurate odd frequency division is avoided.

The PCR in the device is extracted from the adaptation _ field of the MPEG-TS stream through a demultiplexer, and the extraction process of the PCR is realized by hardware. When the second PCR arrives, the hardware gives a PCR interrupt and updates the current PCR and STC counter into a register for software to read.

The PTS/DTS in the device is also extracted by the demultiplexer and is given to the decoder, and when one PTS/DTS is decoded, the PTS/DTS is used as the display/decoding time of the Elementary Stream (ES) data which is then sent into the Input buffer of the decoder, but the ES data which is sent into the buffer is not necessarily decoded immediately at the moment, so the decoder has to store the PTS/DTS into a First-in First-out (FIFO) queue and record the position of the code Stream corresponding to the PTS/DTS, thereby being beneficial to the synchronization of the subsequent decoding and display.

By adopting the device, the system clock can be accurately reconstructed at the decoding end, and when the system clock has deviation, the system clock can be adjusted by software in a short time, so that the system clock is consistent with the encoding end, and an accurate reference clock is further provided for audio and video synchronization.

Example one

Referring to fig. 2, it shows a method for audio and video synchronization processing provided in an embodiment of the present invention, where the method includes:

s101, comparing the current program clock reference PCR value with the current system clock STC value to obtain a first comparison result.

Referring to fig. 3, specifically, step S101 includes steps S1011 to S1014:

and S1011, loading the first PCR value or the reloaded PCR value as a counting initial value into a local STC counter.

It should be noted that, when the first PCR arrives or the PCR needs to be reloaded, the current PCR value is loaded into the local STC counter as the initial count value. The local STC counter also divides the STC _ base and the STC _ extension to count respectively; the STC _ base is a basic value of the STC, is 33 bits higher than the STC, has an initial value of PCR _ base, and then is driven by a 90kHz clock to count up, and is mainly used for comparing with PTS and DTS so as to adjust the audio and video decoding and displaying speed; STC _ extension is an extension value of STC, is 9 bits lower than the STC, has an initial value of PCR _ extension, is driven by a 27MHz clock to count up, and is mainly used for recovering a system clock.

S1012, driving the local STC counter by using the local system clock frequency, and counting STC in an increasing way; the local system clock frequency is N times of the coding system clock frequency, and N is an integer greater than or equal to 1.

Preferably, the local STC counter counts driven by a local 108MHz clock. The clock of the encoding end system is 27MHz, and a 108MHz local clock is adopted to improve the precision of the counter, so that the phase-locked loop is adjusted more accurately through software. STC _ base adds 1 every 1200 clock cycles and STC _ extension adds 1 every 4 clock cycles.

And S1013, when the PCR interruption is detected, reading the current STC value and the current PCR value in the local STC counter.

Specifically, when the next PCR comes, the hardware extracts the current STC value and PCR value into the register for software to query, and gives an interrupt to the PCR at the same time, and the software detects the interrupt, i.e., reads the current STC and PCR value.

S1014, calculating the absolute value of the difference between the current PCR value and the current STC value in the local STC counter; wherein an absolute value of a difference between the current PCR value and the current STC value in the local STC counter is the first comparison result.

Specifically, the absolute value of the difference between the current PCR value and the current STC value is calculated by software, and the calculation expression is as follows: Δ PCR ═ PCR-STC |. And after the software calculates the absolute value of the difference between the current PCR value and the current STC value, the PCR interruption is eliminated.

And S102, when the first comparison result does not meet the preset judgment condition, reconstructing the system clock STC by adjusting the current STC counting step length.

Specifically, referring to fig. 4, step S102 includes steps S1021 to S1024:

and S1021, comparing the first comparison result with a preset judgment condition.

The first comparison result is Δ PCR ═ PCR-STC |, and Δ PCR is divided into two parts, Δ PCR _ base and Δ PCR _ extension. The values of Δ PCR _ base and Δ PCR _ extension are respectively preset with a value range as a preset judgment condition, and preferably, the preset judgment condition is satisfied when Δ PCR _ base is 0 and Δ PCR _ extension is less than or equal to 162.

And S1022, comparing the current PCR value with the current STC value when the first comparison result does not meet a preset judgment condition.

If the condition that the delta PCR _ base is not equal to 0 and the delta PCR _ extension is not greater than 162 is not met, comparing the current PCR value with the current STC value, and judging the speed of the STC counting speed according to the comparison result of the current PCR value and the current STC value so as to correspondingly adjust the STC counting step length, wherein the initial value of the STC counting step length is the initial valueIs 2³⁰。

And S1023, if the current PCR value is larger than the current STC value, increasing the current STC counting step length.

It can be understood that if PCR > STC, it means STC count is too slow, and STC count step needs to be increased, and the STC system clock is adjusted accordingly.

And S1024, if the current PCR value is smaller than the current STC value, reducing the current STC counting step size.

It can be understood that if the PCR < STC, which means that the STC count is too fast, the STC count step needs to be decreased, and the STC system clock is adjusted accordingly.

Specifically, the method further comprises:

It can be understood that if the condition Δ PCR _ base ═ 0 and Δ PCR _ extension ≦ 162 is satisfied, it is determined that the system clock STC is normal, and the system clock does not need to be reconstructed, and therefore, the STC count step size does not need to be adjusted.

It should be noted that, when the demultiplexer performs MPEG-2 TS packet header parsing, PCR information is extracted and sent to the system clock recovery circuit. When each new PCR arrives, the new PCR is compared with a local STC counter to recover and phase-lock the system clock. Therefore, in the process of playing the program, if the local system clock is deviated from the coding system clock, the local system clock can be corrected after the next PCR comes, and accurate system clock reference is provided for the audio and video synchronization processing.

The above process is a process of reconstructing a system clock, and referring to fig. 5, it shows a flowchart of reconstructing a system clock using a PCR and a local STC counter. After the system clock is rebuilt, the audio and video synchronization processing process can be carried out on the basis of rebuilding the system clock, and the audio and video synchronization processing process is as follows:

s103, comparing the absolute value of the difference between the STC basic value STC _ base in the reconstructed system clock STC and the display time stamp PTS value with a preset time interval to obtain a second comparison result; the preset time interval is M times of the inter-frame time of the audio or video, and M is larger than 0 and smaller than 1.

It should be noted that after a new frame in the audio/video comes, the software calculates the absolute value of the difference between STC _ base and PTS, i.e., the value of | STC _ base-PTS |. The audio/video inter-frame time is Δ PTS, and the predetermined time interval is M times the Δ PTS, where M is greater than 0 and less than 1.

Preferably, the value of M is set to 0.5, so that 0.5 times the inter-frame time is Δ PTS/2. And comparing the absolute value of the difference value between the STC _ base value and the PTS value of the audio/video with 0.5 interframe time, namely comparing the | STC _ base-PTS | with the delta PTS/2, wherein the comparison result of the | STC _ base-PTS | and the delta PTS/2 is a second comparison result.

And S104, adjusting the audio or video playing speed according to the second comparison result.

It should be noted that, the judgment and adjustment processes of the audio and video decoding speed are different, the adjustment of the video decoding speed needs to be correspondingly adjusted synchronously according to the structure of the video, and the audio decoding speed can be adjusted only by skipping a sampling point or repeating the previous sampling point according to the decoding speed. Therefore, the step S104 is divided into two cases of decoding video and decoding audio, where S104a1 to S104A3 are the adjustment process for the video decoding speed when decoding video, and S104B1 to S104B3 are the adjustment process for the audio decoding speed when decoding audio.

Referring to fig. 6, specifically, when decoding a video and adjusting the video decoding speed, the specific process includes steps S104a1 to S104 A3:

and S104A1, correspondingly adjusting the decoding speed of the video by using the second comparison result.

It can be understood that the playing speed of the video can be adjusted by correspondingly adjusting the decoding speed of the video according to the second comparison result.

And S104A2, when the absolute value of the difference value between the STC _ base value and the PTS value is smaller than the preset time interval, not adjusting the video decoding speed.

It is understood that if STC _ base-PTS | < Δ PTS/2, it indicates that the video decoding is normal, and no adjustment is required for the video decoding speed.

And S104A3, when the absolute value of the difference between the STC _ base value and the PTS value is larger than or equal to the preset time interval, adjusting the video decoding speed.

It can be understood that | STC _ base-PTS | ≧ Δ PTS/2, it can be known that the video decoding speed is abnormal at this time, and the video decoding speed needs to be adjusted. Since the judgment is carried out according to the absolute value of the difference value between the STC _ base value and the PTS value and the delta PTS/2, the specific situation of the video decoding speed cannot be known, and whether the video decoding speed is high or low cannot be determined, the judgment is carried out by removing the absolute value, and the judgment is that the value of subtracting the PTS from the STC _ base is greater than or equal to the delta PTS/2 or the value of subtracting the STC _ base from the PTS is greater than or equal to the delta PTS/2, so that the video decoding speed is further judged.

For step S104a3, the process of determining and adjusting the video decoding speed specifically includes the following two processes:

when the difference value obtained by subtracting the PTS value from the STC _ base value is larger than or equal to the preset time interval, the video decoding speed is smaller than the preset normal speed, and corresponding synchronous adjustment is carried out according to the structure of the video frame; wherein, the structure of the video frame is respectively as follows: bidirectional predicted frame B frames, intra predicted frame I frames, and unidirectional predicted frame P frames.

It will be appreciated that if STC _ base-PTS ≧ Δ PTS/2, which indicates that video decoding is somewhat slow, the decoder is out of sync and corresponding synchronization adjustments should be made depending on the frame structure.

The video decoding speed is correspondingly and synchronously adjusted according to different video frame structures, so that the process specifically comprises the following steps:

It should be noted that, in video compression coding, all frames are divided into three categories: i (Intra-Prediction), B (Bi-Prediction) and p (Prediction) frames, which are Intra-predicted, Bi-predicted and uni-directional predicted frames, respectively.

And secondly, when the difference value obtained by subtracting the STC _ base value from the PTS value is larger than or equal to the preset time interval, the video decoding speed is larger than the preset normal speed, the last video frame is repeatedly displayed, the video decoding is suspended, and when the STC _ base value is the same as the PTS value, the current image is displayed.

It can be understood that if the PTS-STC _ base ≧ Δ PTS/2, which indicates that video decoding is faster, video decoding can be suspended, the previous frame of picture that has been displayed is repeatedly displayed, by which the decoding speed of video can be slowed down, and when the STC _ base is the same as the PTS, the current frame of picture is displayed again.

Referring to fig. 7, specifically, when audio is decoded and the audio decoding speed is adjusted, the specific process includes steps S104B1 to S104B 3:

and S104B1, correspondingly adjusting the decoding speed of the audio by using the second comparison result.

It is understood that the playing speed of the audio can be adjusted by correspondingly adjusting the decoding speed of the audio according to the second comparison result.

And S104B2, when the absolute value of the difference value between the STC _ base value and the PTS value is smaller than the preset time interval, not adjusting the audio decoding speed.

It is understood that if STC _ base-PTS | < Δ PTS/2, it indicates that audio decoding is normal, and no adjustment is required for the audio decoding speed.

And S104B3, when the absolute value of the difference between the STC _ base value and the PTS value is larger than or equal to the preset time interval, adjusting the audio decoding speed.

It can be understood that if STC _ base-PTS | ≧ Δ PTS/2, it can be known that the audio decoding speed is abnormal at this time, and the audio decoding speed needs to be adjusted. Since the judgment is carried out according to the absolute value of the difference between the STC _ base value and the PTS value and the delta PTS/2, the specific situation of the audio decoding speed cannot be known, and whether the audio decoding speed is high or low cannot be determined, the judgment is carried out by removing the absolute value, and the judgment is that the value of subtracting the PTS from the STC _ base is greater than or equal to the delta PTS/2, or the value of subtracting the STC _ base from the PTS is greater than or equal to the delta PTS/2, so that the audio decoding speed is further judged.

For step S104B3, the process of determining and adjusting the audio decoding speed specifically includes:

The above process is a process of performing audio and video synchronization based on the reconstructed system clock, and referring to fig. 8, it shows a flowchart of performing audio and video synchronization using STC _ base and PTS.

The embodiment of the invention provides an audio and video synchronous processing method, which comprises the steps of adjusting the current STC counting step length to reconstruct a system clock STC when the comparison result of the current PCR value and the current STC value does not meet the preset judgment condition, then adjusting the audio/video playing speed according to the comparison result of the difference absolute value between the STC _ base value and the display time tag PTS value in the reconstructed system clock STC and the preset time interval, and accurately reconstructing a system clock consistent with an encoding end at a decoding end, thereby providing accurate clock reference for the synchronization of audio and video and solving the problem of lip-sound asynchronization in the video playing process.

Example two

Referring to fig. 9, it shows an apparatus 9 for audio and video synchronization processing according to an embodiment of the present invention, the apparatus includes: a comparison module 901 and an adjustment module 902; wherein,

the comparing module 901 is configured to compare the current program clock reference PCR value with the current system clock STC value to obtain a first comparison result;

the adjusting module 902 is configured to, when the first comparison result does not satisfy the preset determination condition, reestablish the system time clock STC by adjusting the current STC count step length;

the comparing module 901 is further configured to compare an absolute value of a difference between an STC basis value STC _ base and a display time stamp PTS value in the reconstructed system clock STC with a preset time interval, so as to obtain a second comparison result; the preset time interval is M times of the inter-frame time of the audio or video, and M is greater than 0 and smaller than 1;

the adjusting module 902 is further configured to adjust the audio or video playing speed according to the second comparison result.

Further, referring to fig. 10, the apparatus further includes: a loading module 903, a counting module 904, a reading module 905 and a calculating module 906; wherein,

the loading module 903 is configured to load the first PCR value or the reloaded PCR value as a count initial value into a local STC counter;

the counting module 904 is configured to drive the local STC counter by using a local system clock frequency, so as to count up the STC; the local system clock frequency is N times of the coding system clock frequency, and N is an integer greater than or equal to 1;

the reading module 905 is configured to read a current STC value and the current PCR value in the local STC counter when detecting that the PCR is interrupted;

the calculating module 906 is configured to calculate an absolute value of a difference between the current PCR value and the current STC value in the local STC counter; wherein an absolute value of a difference between the current PCR value and the current STC value in the local STC counter is the first comparison result.

Further, the comparing module 901 is configured to compare the first comparison result with a preset judgment condition;

the adjusting module 902 is configured to compare the current PCR value with the current STC value when the first comparison result does not satisfy a preset determination condition;

Further, referring to fig. 10, the apparatus further includes: a holding module 907; wherein,

the holding module 907 is configured to, when an absolute value of a difference between the current PCR value and the current STC value meets a preset determination condition, keep an STC count step unchanged.

Further, the adjusting module 902 is configured to perform corresponding adjustment on the decoding speed of the video by using the second comparison result;

Further, the adjusting module 902 is configured to, when a difference obtained by subtracting the PTS value from the STC _ base value is greater than or equal to the preset time interval, perform corresponding synchronization adjustment according to the structure of the video frame, where the video decoding speed is less than a preset normal speed; wherein, the structure of the video frame is respectively as follows: bidirectional predicted frame B frame, intra-frame predicted frame I frame and unidirectional predicted frame P frame;

Further, the adjusting module 902 is configured to discard the current frame, decode the next frame, and simultaneously display the previous frame when the video frame is a B frame;

Further, the adjusting module 902 is configured to perform corresponding adjustment on the decoding speed of the audio by using the second comparison result;

Further, the adjusting module 902 is configured to skip a sampling point when a difference obtained by subtracting the PTS value from the STC _ base value is greater than or equal to the preset time interval, and the audio decoding speed is less than a preset normal speed;

The embodiment of the invention provides an audio and video synchronous processing device, which is characterized in that when the comparison result of a current PCR value and a current STC value does not meet a preset judgment condition, the current STC counting step length is adjusted to reconstruct a system clock STC, then the audio/video playing speed is adjusted according to the comparison result of the absolute value of the difference value of an STC _ base value and a display time tag PTS value in the reconstructed system clock STC and a preset time interval, and a system clock consistent with an encoding end is accurately reconstructed at a decoding end, so that accurate clock reference is provided for the synchronization of audio and video, and the problem of lip-sound asynchronization in the video playing process is solved.

EXAMPLE III

Referring to fig. 11, a device shown in fig. 9 is a hardware implementation device related to reconstruction of a system clock and audio/video synchronization processing provided in an embodiment of the present invention, where the hardware implementation device may specifically include most 3: a CPU, a demultiplexer demux and a decoder.

The demultiplexer demux is divided into a TS _ ph _ get module and a PCR _ recovery module, wherein the TS _ ph _ get module analyzes a packet header, extracts a PCR _ flag and a PCR value from an adaptive area of an MPEG2-TS stream, and sends the PCR _ flag and the PCR value to the PCR _ recovery module to reconstruct a system clock.

The PCR _ recovery module stores the PCR value, and particularly realizes that when a local STC counter has deviation in a local system clock, the STC counter can be adjusted by adjusting the STC step length through the CPU so as to be basically consistent with the current PCR. The PCR _ recovery module sends a PCR _ interrupt signal to the CPU, the PCR _ interrupt signal indicates that the PCR _ recovery module extracts a PCR interrupt and sends the PCR interrupt to the CPU, the CPU queries STC _ regs/PCR _ regs after receiving the interrupt, software calculates a difference between STC and PCR, adjusts STC counting step length according to the difference, and feeds the STC counting step length back to the PCR _ recovery module.

The decoder is divided into an FIFO module, an ES data buffer, a decoding module and a display module.

When a new frame of image is started, the demultiplexer demux writes the PTS/DTS corresponding to the frame of image into the FIFO module, and when a frame of image is decoded, reads out the corresponding DTS from the FIFO and compares the DTS with the STC, and when the two are equal, decoding can be started in the decoding module. Meanwhile, the demultiplexer demux writes the ES data into an ES data buffer, and the ES data buffer sends the ES data to a decoding module for decoding. Before each display unit starts to display, the corresponding PTS is compared with the STC, and when the PTS and the STC are equal, the display can be started in the display module. The ES data is an Elementary Stream (ES) containing continuous streams of video, audio, or data.

It should be noted that not every frame of image has PTS and DTS, a PES packet may have multiple frames of images, only the first frame of image has PTS, and the other frames of the PES packet have no PTS. If a frame image has no PTS, an increment can be added on the basis of the previous PTS to obtain the PTS corresponding to the frame image, the PTS corresponding to the frame image is sent into the FIFO for management, and audio and video synchronization processing is carried out according to the method. PES (packet Elementary stream) is a packetized Elementary stream, and is a packetized Elementary stream PES stream formed by dividing an Elementary stream ES stream into packets with different lengths as required and adding a packet header.

The embodiment of the invention provides a hardware implementation device of an audio and video synchronous processing device, which is characterized in that when the comparison result of a current PCR value and a current STC value does not meet a preset judgment condition, the current STC counting step length is adjusted to reconstruct a system clock STC, then the audio/video playing speed is adjusted according to the comparison result of the absolute value of the difference value between the STC _ base value in the reconstructed system clock STC and a display time tag PTS value and a preset time interval, and a system clock consistent with a coding end is accurately reconstructed at a decoding end, so that accurate clock reference is provided for audio and video synchronization, and the problem of lip sound asynchrony in the video playing process is solved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.

Claims

1. A method for audio and video synchronization processing is characterized by comprising the following steps:

adjusting the audio or video playing speed according to the second comparison result;

when the video is decoded, the decoding speed of the video is correspondingly adjusted by using the second comparison result; when the audio is decoded, the decoding speed of the audio is correspondingly adjusted by using the second comparison result.

2. The method of claim 1, wherein comparing the current program clock reference PCR value with the current system clock STC value to obtain the first comparison result comprises:

3. The method of claim 2, wherein when the first comparison result does not satisfy the predetermined judgment condition, reconstructing the system clock STC by adjusting the current STC count step size comprises:

comparing the first comparison result with a preset judgment condition;

4. The method of claim 1, further comprising:

5. The method of claim 1, wherein said adjusting the video playback speed according to the second comparison result when decoding the video comprises:

6. The method of claim 5, wherein the adjusting the video decoding speed when the absolute value of the difference between the STC _ base value and the PTS value is greater than or equal to the predetermined time interval comprises:

7. The method of claim 6, wherein when the difference obtained by subtracting the PTS value from the STC _ base value is greater than or equal to a predetermined time interval, and the video decoding speed is less than a predetermined normal speed, performing corresponding synchronization adjustment according to the structure of the video frame comprises:

8. The method of claim 1, wherein adjusting the audio playback speed according to the second comparison result when decoding audio comprises:

9. The method of claim 8, wherein the adjusting the audio decoding speed when the absolute value of the difference between the STC _ base value and the PTS value is greater than or equal to the preset time interval comprises:

10. An apparatus for audio and video synchronization processing, the apparatus comprising: a comparison module and an adjustment module; wherein,

the adjusting module is further configured to adjust the audio or video playing speed according to the second comparison result; when the video is decoded, the adjusting module utilizes the second comparison result to correspondingly adjust the decoding speed of the video; when the audio is decoded, the adjusting module uses the second comparison result to correspondingly adjust the decoding speed of the audio.

11. The apparatus of claim 10, further comprising: the device comprises a loading module, a counting module, a reading module and a calculating module; wherein,

12. The apparatus of claim 11,

the comparison module is used for comparing the first comparison result with a preset judgment condition;

13. The apparatus of claim 10, further comprising: a holding module; wherein,

14. The apparatus of claim 10,

the adjusting module is configured to not adjust the video decoding speed when the absolute value of the difference between the STC _ base value and the PTS value is smaller than the preset time interval;

15. The apparatus of claim 14,

the adjusting module is configured to perform corresponding synchronization adjustment according to the structure of the video frame, when a difference obtained by subtracting the PTS value from the STC _ base value is greater than or equal to the preset time interval, and the video decoding speed is lower than a preset normal speed; wherein, the structure of the video frame is respectively as follows: bidirectional predicted frame B frame, intra-frame predicted frame I frame and unidirectional predicted frame P frame;

16. The apparatus of claim 15,

the adjusting module is used for discarding the current frame, decoding the next frame and displaying the previous frame when the video frame is a B frame;

17. The apparatus of claim 10,

the adjusting module is used for not adjusting the audio decoding speed when the absolute value of the difference value between the STC _ base value and the PTS value is smaller than the preset time interval;

18. The apparatus of claim 17,

the adjusting module is used for skipping a sampling point when the audio decoding speed is less than a preset normal speed when the difference value obtained by subtracting the PTS value from the STC _ base value is greater than or equal to the preset time interval;