CN108495164B - Audio and video synchronization processing method and device, computer device and storage medium - Google Patents
Audio and video synchronization processing method and device, computer device and storage medium Download PDFInfo
- Publication number
- CN108495164B CN108495164B CN201810310294.6A CN201810310294A CN108495164B CN 108495164 B CN108495164 B CN 108495164B CN 201810310294 A CN201810310294 A CN 201810310294A CN 108495164 B CN108495164 B CN 108495164B
- Authority
- CN
- China
- Prior art keywords
- frame
- video
- audio
- difference value
- video frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4305—Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Abstract
The invention discloses an audio and video synchronous processing method and device, a computer device and a storage medium, wherein the method comprises the steps of determining that a decoder is in an initialization state, acquiring data of an effective video code stream, outputting a decoded video frame, judging whether the decoded video frame is normal or not, if so, outputting the video frame and the audio frame, generating a first difference value according to the audio frame, generating a second difference value according to the video frame, judging whether the first difference value and the second difference value are normal or not, if so, outputting a first playing signal, otherwise, deleting the data of an abnormal video frame and the data of an abnormal audio frame, and resetting a player. The device is used for realizing the audio and video synchronization processing method. The invention also provides a computer device and a storage medium for realizing the method. The invention can not only eliminate the asynchronous phenomenon of the Kanton audio and video, but also realize the purpose of smoother video.
Description
Technical Field
The invention relates to the technical field of audio and video processing, in particular to an audio and video synchronization processing method and an audio and video synchronization processing device for realizing the method, and further relates to a computer device and a storage medium applying the method.
Background
Watching video is an important way for people to gain information and entertain. However, when a video file is damaged due to manufacturing, transcoding, network transmission, and the like, the playing of the audio/video file is abnormal, typically, the phenomena of screen splash, blockage, audio/video asynchronism, and the like occur during playback.
At present, in the industry, a player has a method for recovering a splash screen frame caused by decoding a damaged code stream, the splash screen video frame can be recovered to a normal video frame to be output and displayed to a certain extent by a certain method, the splash screen frame can be recovered to a normal frame to a certain extent, however, the method consumes a long time, meanwhile, the seriously damaged code stream cannot be recovered, and the jam and the splash screen still exist.
In addition, some players discard abnormal audio and video frames of the video frames output by decoding after checking the code stream in a decoder, but the phenomenon of pause and audio and video asynchrony of audio output and video play can be caused by the fact that no video frame is refreshed and sent to display due to discarding of the video frames of the video screens.
Disclosure of Invention
The invention aims to provide an audio and video synchronization processing method for performing audio and video synchronization alignment and adjusting a player clock by deleting an audio frame or a video frame.
A second object of the present invention is to provide an audio/video synchronization processing apparatus that can not only eliminate the video asynchronism phenomenon of katon audio but also achieve smoother video.
The third objective of the present invention is to provide a computer device capable of implementing the above audio/video synchronization processing method.
A fourth object of the present invention is to provide a storage medium capable of implementing the above audio/video synchronization processing method.
In order to achieve the first object, the audio and video synchronization processing method provided by the invention comprises the steps of determining that a decoder is in an initialization state, acquiring data of an effective video code stream, and outputting a decoded video frame; judging whether the decoded video frame has no screen-splash video frame for a preset number of continuous times, if so, outputting the video frame and the audio frame, otherwise, determining that the video frame is abnormal, and resetting the decoder; generating a first difference value according to the audio frame and a second difference value according to the video frame, wherein the first difference value is the difference value between the time stamp of the nth frame of audio frame and the first clock value, and the second difference value is the difference value between the time stamp of the nth frame of video frame and the time stamp of the (n-1) th frame of video frame; judging whether the first difference value and the second difference value are normal: if the difference value between the timestamp of the detected first frame audio frame and the first clock value is greater than the preset first time when the first frame audio frame is transmitted and displayed, determining that the first difference value is abnormal; or if the difference value between the time stamp of the second frame video frame and the time stamp of the first frame video frame is detected to be larger than a preset value when the second frame video frame is sent and displayed, determining that the second difference value is abnormal; if the first difference value and the second difference value are judged to be normal, outputting a first playing signal, if the judgment result is negative, deleting the data of the abnormal video frame and the data of the abnormal audio frame, and resetting the player.
According to the scheme, the audio and video synchronous processing method provided by the invention has the advantages that the decoder reads the effective code stream for decoding, detects the decoded audio and video frames and deletes the abnormal audio frames or video frames to ensure that the audio and video frames for display are normal frames.
In addition, the change of the audio/video frame timestamp can be detected in the process of playing the display stream, when the abnormal timestamp change is detected, the audio or video frame is deleted to carry out display alignment operation, and the clock of the player is adjusted to output the playing signal.
The method comprises the following steps that if screen-splash video frames which are continuously preset times do not appear in decoded video frames, the video frames are determined to be normal; if an odd video frame occurs a preset number of consecutive times in the decoded video frame, it is determined that the video frame is abnormal, and the decoder is reset.
It can be seen that the preset number of the splash-screen frames are set to occur continuously in order to decode the effective code stream as much as possible, for example, the preset number of the splash-screen frames may be 3 frames, 4 frames or 5 frames, and meanwhile, if there are more error frames, the frequent reset of the decoder to cause the delay of decoding time to increase and cause the pause can be avoided.
The method comprises the following steps that when the first difference value is determined to be abnormal, video frame data between a first clock value and a time stamp of a first frame of audio frame are deleted, wherein the first clock value is the clock value of a player when the first frame of audio frame is sent and displayed; and when the second difference value is determined to be abnormal, deleting the audio frame data between the time stamp of the first frame video frame and the time stamp of the second frame video frame.
Therefore, whether the audio frame and the video frame of the nth frame are normally played and displayed is judged according to the difference value between the time stamp of the audio frame of the nth frame and the corresponding clock value and the difference value between the time stamp of the video frame of the nth frame and the time stamp of the video frame of the (n-1) th frame.
If it is detected that the first difference is greater than a preset time when the nth frame of audio frame is displayed, for example, the preset time may be 100ms, the audio frame is considered to be discontinuous, and video frame data between the corresponding clock value and the timestamp of the nth frame of audio frame is deleted.
If the second difference is detected to be greater than a preset value when the nth Frame of video Frame is displayed, for example, the preset value is 2 × 1/Frame _ Rate, where the Frame _ Rate is the Frame Rate of the film source, the video Frame is considered to be discontinuous, and the audio Frame data between the timestamp of the nth-1 Frame of video Frame and the timestamp of the nth Frame of video Frame is deleted.
Further, after determining to delete the video frame data, the clock signal of the player is reset according to the timestamp of the first frame of audio frame, and the second playing signal is output.
It can be seen that, if the first difference is detected to be abnormal when the nth audio frame is sent and displayed, after the audio frame data is determined to be deleted, the player clock is reset by using the timestamp of the nth audio frame and then played.
Or if the second difference is detected to be abnormal when the nth frame of video frame is sent and displayed, resetting the player clock by using the timestamp of the nth frame of video frame and then playing after the video frame data is determined to be deleted, thereby eliminating the pause and the audio-video asynchronization.
In order to achieve the second objective, the present invention provides an audio and video synchronization processing apparatus, which includes a decoding module, configured to obtain data of an effective video code stream, and output a decoded video frame; the judging module is used for judging whether the decoded video frame has no screen-splash video frame for continuous preset times, if so, outputting the video frame and the audio frame, otherwise, determining that the video frame is abnormal, and resetting the decoder; the synchronization module generates a first difference value according to the audio frame and a second difference value according to the video frame, wherein the first difference value is the difference value between the time stamp of the nth video frame and the time stamp of the nth video frame, and the second difference value is the difference value between the time stamp of the nth video frame and the time stamp of the (n-1) th video frame; the synchronization module is further configured to determine whether the first difference and the second difference are normal: if the difference value between the timestamp of the detected first frame audio frame and the first clock value is greater than the preset first time when the first frame audio frame is transmitted and displayed, determining that the first difference value is abnormal; or if the difference value between the time stamp of the second frame video frame and the time stamp of the first frame video frame is detected to be larger than a preset value when the second frame video frame is sent and displayed, determining that the second difference value is abnormal; if the first difference value and the second difference value are judged to be normal, outputting a first playing signal, if the judgment result is negative, deleting the data of the abnormal video frame and the data of the abnormal audio frame, and resetting the player.
The further scheme is that if the screen-splash video frames with continuous preset times do not appear in the decoded video code stream, the video frames are determined to be normal; and if the video frames with the screen splash continuously occur for a preset number of times in the decoded video code stream, determining that the video frames are abnormal, and resetting the decoder.
The method comprises the following steps that when the first difference value is determined to be abnormal, video frame data between a first clock value and a time stamp of a first frame of audio frame are deleted, wherein the first clock value is the clock value of a player when the first frame of audio frame is sent and displayed; and when the second difference value is determined to be abnormal, deleting the audio frame data between the time stamp of the first frame video frame and the time stamp of the second frame video frame.
Further, after determining to delete the video frame data, the clock signal of the player is reset according to the timestamp of the first frame of audio frame, and the second playing signal is output.
According to the scheme, the audio and video synchronization processing device reads the effective code stream through the decoding module to decode, and the synchronization module detects the decoded audio and video frames and deletes abnormal audio frames or video frames to ensure that the audio and video frames for display are normal frames.
In addition, the change of the audio/video frame timestamp can be detected in the process of playing the display stream, when the abnormal timestamp change is detected, the audio or video frame is deleted to carry out display alignment operation, and the clock of the player is adjusted to output the playing signal.
In order to achieve the third object, the invention provides a computer device comprising a processor, which is used for implementing the steps of the audio and video synchronization processing method according to the invention when executing the computer program stored in the memory.
In order to achieve the fourth object, the present invention provides a storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the audio and video synchronization processing method according to the present invention are realized.
Drawings
Fig. 1 is a flow chart of an embodiment of an audio and video synchronization processing method according to the present invention.
Fig. 2 is a schematic diagram of playing audio and video by a damaged film source in the embodiment of the audio and video synchronization processing method of the present invention.
Fig. 3 is a schematic block diagram of an embodiment of the audio and video synchronization processing apparatus of the present invention.
The invention is further explained with reference to the drawings and the embodiments.
Detailed Description
The audio and video synchronous processing method is applied to terminals such as display equipment or video playing equipment, for example, the method can be applied to a digital video conversion box, and can effectively solve the abnormal phenomena of asynchronous video and the like of the dot-screen Kanton audio caused by a damaged film source. The invention relates to an audio and video synchronization processing device which is applied to a display terminal and is used for realizing an audio and video synchronization processing method. Moreover, the invention also provides a computer device, which comprises a processor, wherein the processor can execute the instruction of the application program, so that each step of the audio and video synchronization processing method is realized. The storage medium provided by the invention stores a computer program, and the computer program realizes each step of the audio and video synchronization processing method when being executed by a processor.
The embodiment of the audio and video synchronization processing method comprises the following steps:
as shown in fig. 1, when processing the audio/video data, the audio/video synchronization processing method of the present invention first executes step S1 to obtain data of an effective video code stream and output a decoded video frame. And after the decoder is determined to be in the initialization state, the decoder is used for acquiring the data of the effective video code stream and decoding and outputting the data. Then, step S2 is executed to determine whether the decoded video frame is normal. And if the screen-splash video frames which are continuously preset times do not appear in the decoded video frames, determining that the video frames are normal. If the splash-screen video frames of continuous preset times appear in the decoded video code stream, determining that the video frames are abnormal, resetting a decoder to an initialization state, emptying a reference queue and searching the next IDR frame for Decoding, wherein the IDR (instantaneous Decoding refresh) is refreshed immediately, and the IDR frame is refreshed immediately, so that errors are not propagated, and a new sequence is recalculated from the IDR frame to start encoding. It can be seen that the preset number of the splash-screen frames are set to occur continuously in order to decode the effective code stream as much as possible, for example, the preset number of the splash-screen frames may be 3 frames, 4 frames or 5 frames, and meanwhile, if there are more error frames, the frequent reset of the decoder to cause the delay of decoding time to increase and cause the pause can be avoided.
Next, if it is determined that the decoded video frame is normal, step S3 is executed to generate a first difference value according to the audio frame and a second difference value according to the video frame. The first difference is the difference between the time stamp of the nth frame of audio frame and the first clock value, and the second difference is the difference between the time stamp of the nth frame of video frame and the time stamp of the (n-1) th frame of video frame. In this embodiment, the first clock value is the clock value of the player 4 when the nth frame of audio frame is transmitted and displayed.
Then, step S4 is executed to determine whether the first difference and the second difference are normal. If yes, step S5 is executed to output a playback signal to the player. And if the difference value between the timestamp of the detected first frame audio frame and the first clock value is greater than the preset first time when the first frame audio frame is displayed, determining that the first difference value is abnormal.
Or, if the difference between the timestamp of the second frame video frame and the timestamp of the first frame video frame is detected to be larger than a preset value when the second frame video frame is sent and displayed, determining that the second difference is abnormal.
Then, if it is determined that the first difference or the second difference is not normal, step S6 is performed to delete the data of the abnormal video frame and the data of the abnormal audio frame according to the time stamp. When the first difference value is determined to be abnormal, deleting video frame data between the first clock value and the time stamp of the first frame of audio frame; and when the second difference value is determined to be abnormal, deleting the audio frame data between the time stamp of the first frame video frame and the time stamp of the second frame video frame.
Specifically, whether the audio frame and the video frame of the nth frame are played and displayed normally is judged according to the difference value between the time stamp of the audio frame of the nth frame and the corresponding clock value and the difference value between the time stamp of the video frame of the nth frame and the time stamp of the video frame of the (n-1) th frame.
And if the first difference value is detected to be larger than the preset time when the nth frame of audio frame is sent and displayed, the audio frame is considered to be discontinuous, and video frame data between the corresponding clock value and the time stamp of the nth frame of audio frame is deleted. Of course, in order to improve the accuracy and efficiency of the detection, it is necessary to determine whether the first difference is within the normal range by whether a preset time is reached, and preferably, the preset time may be 100 ms.
Specifically, the preset time may be set by the user according to the actual application, and the user may set the preset time according to the specific use environment and the operation performance of the display terminal, and for the setting of the time, the time in the above embodiment is set to be 100ms, and may also be set to be other values such as 100ms, 90ms, and 110ms, where in this embodiment, the optimal preset time is 100 ms.
If the second difference is detected to be greater than a preset value when the nth Frame of video Frame is displayed, for example, the preset value is 2 × 1/Frame _ Rate, where the Frame _ Rate is the Frame Rate of the film source, the video Frame is considered to be discontinuous, and the audio Frame data between the timestamp of the nth-1 Frame of video Frame and the timestamp of the nth Frame of video Frame is deleted.
After determining to delete the video frame data, step S7 is executed to reset the clock signal of the player 4, wherein the clock signal of the player 4 is reset according to the timestamp of the first audio frame, and the playing signal is output to the player 4.
For example, referring to fig. 2, a1, a2, a3 are audio frames which are synchronously output and appear as normal play in a one-to-one correspondence manner, p1, p2 and p3 are video frames which are synchronously output and appear as normal play in a one-to-one correspondence manner, when the Time stamp of p4 and the Time stamp of p3 are detected to be greater than 2 × 1/Frame _ Rate when the display is presented p4, audio Frame data between the Time stamp of p3 and the Time stamp of p4 are deleted to ensure the display alignment of audio and video frames, and the player 4 clock is reset by the Time stamp of p4, so that the audio and video play is directly jumped from (Time-a, a3 and p3) to (Time-B, a4 and p4) and the card pause and audio and video asynchronization are eliminated.
It can be seen that, if the first difference is detected to be abnormal when the nth audio frame is sent and displayed, after the audio frame data is determined to be deleted, the clock of the player 4 is reset by using the timestamp of the nth audio frame and then played.
Or, if the second difference is detected to be abnormal when the nth frame of video frame is sent and displayed, after the video frame data is determined to be deleted, the clock of the player 4 is reset by using the timestamp of the nth frame of video frame and then played, so that the pause, the audio and the video asynchronization are eliminated.
Therefore, the audio and video synchronization processing method of the invention can detect the decoded audio and video frames and delete the abnormal audio frames or video frames by reading the effective code stream in the decoder for decoding, thereby ensuring that the audio and video frames for display are all normal frames.
In addition, the change of the audio/video frame timestamp can be detected in the process of playing the display stream, when the abnormal timestamp change is detected, the audio or video frame is deleted to carry out display alignment operation, and the clock of the player 4 is adjusted and then a playing signal is output.
The embodiment of the audio and video synchronization processing device comprises:
referring to fig. 3, the audio and video synchronization processing apparatus of the present invention includes a decoding module 1, a determining module 2, a synchronization module 3, and a player 4.
The decoding module 1 is configured to obtain data of an effective video code stream and output a decoded video frame. Preferably, the decoding module 1 includes a decoder, the decoder can decode and restore the digital video and audio data stream into an analog video and audio signal, and after determining that the decoder is in the initialization state, the decoder is used for acquiring data of an effective video code stream and decoding and outputting the data.
The judging module 2 is used for judging whether the decoded video frame is normal, and if so, outputting the video frame and the audio frame. And if the screen-splash video frames which are continuously preset times do not appear in the decoded video frames, determining that the video frames are normal. And if the video frames with the screen splash continuously occur for a preset number of times in the decoded video frames, determining that the video frames are abnormal, resetting the decoder to an initialization state, emptying the reference queue and searching the next IDR frame for decoding.
The synchronization module 3 generates a first difference value according to the audio frame and a second difference value according to the video frame, the synchronization module 3 is configured to determine whether the first difference value and the second difference value are normal, if the determination result is yes, output a playing signal to the player 4, if the determination result is no, delete data of the abnormal video frame and data of the abnormal audio frame, and reset the player 4. The first difference is the difference between the time stamp of the nth frame of audio frame and the first clock value, and the second difference is the difference between the time stamp of the nth frame of video frame and the time stamp of the (n-1) th frame of video frame.
Therefore, the audio and video synchronization processing device of the invention reads the effective code stream through the decoding module 1 for decoding, and the synchronization module 3 detects the decoded audio and video frames and deletes the abnormal audio frames or video frames to ensure that the audio and video frames for display are all normal frames.
In addition, the change of the audio/video frame timestamp can be detected in the process of playing the display stream, when the abnormal timestamp change is detected, the audio or video frame is deleted to carry out display alignment operation, and the clock of the player 4 is adjusted and then a playing signal is output.
The embodiment of the computer device comprises:
the computer device of the embodiment includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as an information processing program for implementing the information processing method. And when the processor executes the computer program, the steps of the audio and video synchronization processing method are realized.
For example, a computer program may be partitioned into one or more modules that are stored in a memory and executed by a processor to implement the modules of the present invention. One or more of the modules may be a series of computer program instruction segments capable of performing certain functions, which are used to describe the execution of the computer program in the terminal device.
It should be noted that the terminal device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal device may include, but is not limited to, a processor, a memory. It will be understood by those skilled in the art that the schematic diagram of the present invention is merely an example of a terminal device, and does not constitute a limitation of the terminal device, and may include more or less components than those shown, or combine some components, or different components, for example, the terminal device may further include an input-output device, a network access device, a bus, etc.
The Processor may be a Central Processing Unit (CPU), or may be other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being the control center of the terminal device and connecting the various parts of the entire terminal device using various interfaces and lines.
The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the terminal device by running or executing the computer programs and/or modules stored in the memory and invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
Storage medium embodiments:
the storage medium of the present invention may be any form of storage medium that can be read by a processor of a computer device, including but not limited to a nonvolatile memory, a volatile memory, a ferroelectric memory, and the like, and the storage medium has a computer program stored thereon, and when the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-described respective audio and video synchronization processing methods can be implemented.
Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U.S. disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution media, and the like. It should be noted that the computer readable medium may contain other components which may be suitably increased or decreased as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, in accordance with legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunications signals.
It should be noted that the above is only a preferred embodiment of the present invention, but the design concept of the present invention is not limited thereto, and any insubstantial modifications made by using the design concept also fall within the protection scope of the present invention.
Claims (8)
1. The audio and video synchronization processing method is characterized by comprising the following steps:
determining that a decoder is in an initialization state, acquiring data of an effective video code stream, and outputting a decoded video frame;
judging whether the decoded video frame has no screen-splash video frame for a preset number of continuous times, if so, outputting the video frame and the audio frame, otherwise, determining that the video frame is abnormal, and resetting the decoder;
generating a first difference value according to the audio frame, and generating a second difference value according to the video frame, wherein the first difference value is a difference value between a timestamp of the nth frame of audio frame and a first clock value, and the second difference value is a difference value between the timestamp of the nth frame of video frame and a timestamp of the (n-1) th frame of video frame, and the first clock value is a clock value of a player when the nth frame of audio frame is sent and displayed;
judging whether the first difference value and the second difference value are normal: if the detected difference value between the time stamp of the first frame audio frame and the first clock value is greater than the preset first time when the first frame audio frame is sent and displayed, determining that the first difference value is abnormal; or if the difference value between the time stamp of the second frame video frame and the time stamp of the first frame video frame is detected to be larger than a preset value when the second frame video frame is sent and displayed, determining that the second difference value is abnormal;
if the first difference value and the second difference value are judged to be normal, outputting a first playing signal, if the judgment result is negative, deleting the data of the abnormal video frame and the data of the abnormal audio frame, and resetting the player.
2. The audio-video synchronization processing method according to claim 1, characterized in that:
deleting video frame data between the first clock value and a timestamp of the first frame of audio frame when the first difference value is determined to be abnormal;
and when the second difference value is determined to be abnormal, deleting the audio frame data between the time stamp of the first frame video frame and the time stamp of the second frame video frame.
3. The audio-video synchronization processing method according to claim 2, wherein deleting the video frame data between the first clock value and the time stamp of the first frame audio frame comprises:
and after the video frame data are determined to be deleted, resetting a clock signal of the player according to the timestamp of the first frame of audio frame, and outputting a second playing signal.
4. Audio-video synchronization processing apparatus, characterized in that the apparatus comprises:
the decoding module is used for determining that the decoder is in an initialization state, acquiring data of an effective video code stream and outputting a decoded video frame;
the judging module is used for judging whether the decoded video frame has no screen-splash video frame for continuous preset times, if so, the video frame and the audio frame are output, otherwise, the video frame is determined to be abnormal, and the decoder is reset;
the synchronization module generates a first difference value according to the audio frame and a second difference value according to the video frame, wherein the first difference value is a difference value between a timestamp of the nth frame of audio frame and a first clock value, the second difference value is a difference value between the timestamp of the nth frame of video frame and a timestamp of the (n-1) th frame of video frame, and the first clock value is a clock value of a player when the nth frame of audio frame is sent and displayed;
the synchronization module is further configured to determine whether the first difference and the second difference are normal: if the detected difference value between the time stamp of the first frame audio frame and the first clock value is greater than the preset first time when the first frame audio frame is sent and displayed, determining that the first difference value is abnormal; or if the difference value between the time stamp of the second frame video frame and the time stamp of the first frame video frame is detected to be larger than a preset value when the second frame video frame is sent and displayed, determining that the second difference value is abnormal;
if the first difference value and the second difference value are judged to be normal, outputting a first playing signal, if the judgment result is negative, deleting the data of the abnormal video frame and the data of the abnormal audio frame, and resetting the player.
5. The audio-video synchronization processing apparatus according to claim 4, wherein:
deleting video frame data between the first clock value and a timestamp of the first frame of audio frame when the first difference value is determined to be abnormal;
and when the second difference value is determined to be abnormal, deleting the audio frame data between the time stamp of the first frame video frame and the time stamp of the second frame video frame.
6. The audio-video synchronization processing apparatus according to claim 5, wherein the synchronization module deletes the video frame data between the first clock value and the time stamp of the first frame audio frame, including:
and after the video frame data are determined to be deleted, resetting a clock signal of the player according to the timestamp of the first frame of audio frame, and outputting a second playing signal.
7. A computer arrangement comprising a processor and a memory, the memory storing a computer program which, when executed by the processor, carries out the steps of the audio-video synchronization processing method according to any one of claims 1 to 3.
8. A storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the steps of the audio-video synchronization processing method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810310294.6A CN108495164B (en) | 2018-04-09 | 2018-04-09 | Audio and video synchronization processing method and device, computer device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810310294.6A CN108495164B (en) | 2018-04-09 | 2018-04-09 | Audio and video synchronization processing method and device, computer device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108495164A CN108495164A (en) | 2018-09-04 |
| CN108495164B true CN108495164B (en) | 2021-01-29 |
Family
ID=63315051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810310294.6A Active CN108495164B (en) | 2018-04-09 | 2018-04-09 | Audio and video synchronization processing method and device, computer device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108495164B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109089130B (en) * | 2018-09-18 | 2020-05-22 | 网宿科技股份有限公司 | Method and device for adjusting timestamp of live video |
| CN111131874B (en) * | 2018-11-01 | 2021-03-16 | 珠海格力电器股份有限公司 | Method, equipment and computer storage medium for solving problem of playing jam of H.265 code stream random access point |
| CN109348247B (en) * | 2018-11-23 | 2021-03-30 | 广州酷狗计算机科技有限公司 | Method and device for determining audio and video playing time stamp and storage medium |
| CN109905763B (en) * | 2019-03-20 | 2021-09-10 | 四川长虹电器股份有限公司 | Method for optimizing audio playing pause of player |
| TWI743774B (en) * | 2019-07-18 | 2021-10-21 | 瑞昱半導體股份有限公司 | Method for synchronizing audio and video and related apparatus |
| CN112565016B (en) * | 2020-12-04 | 2022-09-30 | 北京奇艺世纪科技有限公司 | Positioning method, system and device for abnormal time delay, electronic equipment and storage medium |
| CN112770165B (en) * | 2020-12-28 | 2022-11-18 | 杭州电子科技大学 | Distributed synchronization method for audio and video streams |
| CN113225617A (en) * | 2021-04-28 | 2021-08-06 | 臻迪科技股份有限公司 | Playing video processing method and device and electronic equipment |
| CN114121050A (en) * | 2021-11-30 | 2022-03-01 | 云知声智能科技股份有限公司 | Audio playing method and device, electronic equipment and storage medium |
| CN114143595A (en) * | 2021-12-08 | 2022-03-04 | 珠海豹趣科技有限公司 | Video wallpaper playing method and device, electronic equipment and readable storage medium |
| CN114390314A (en) * | 2021-12-30 | 2022-04-22 | 咪咕文化科技有限公司 | Variable frame rate audio and video processing method, equipment and storage medium |
| CN114666638B (en) * | 2022-03-11 | 2023-11-28 | 平安国际智慧城市科技股份有限公司 | Audio and video data synchronization method, device, equipment and storage medium |
| CN115361574B (en) * | 2022-08-15 | 2023-09-15 | 广州市奥威亚电子科技有限公司 | Receiving end video processing method, device, equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11191286A (en) * | 1997-07-28 | 1999-07-13 | Sony Electron Inc | Synchronizing method for audio and video signals and digital video processor |
| CN101827271A (en) * | 2009-03-04 | 2010-09-08 | 联芯科技有限公司 | Audio and video synchronized method and device as well as data receiving terminal |
| CN102075806A (en) * | 2011-01-26 | 2011-05-25 | 四川长虹电器股份有限公司 | Audio and video synchronization method of digital television |
| CN103607664A (en) * | 2013-11-22 | 2014-02-26 | 天津市北海通信技术有限公司 | Audio and video synchronization method of embedded multimedia playing system |
| CN103873952A (en) * | 2012-12-18 | 2014-06-18 | 风网科技（北京）有限公司 | High-efficiency video playing frame loss control system and method |
| CN104202611A (en) * | 2014-09-01 | 2014-12-10 | 赛特斯信息科技股份有限公司 | Method of realizing detection on broken screen defect in video file based on video decoder |
| CN106612452A (en) * | 2015-10-22 | 2017-05-03 | 深圳市中兴微电子技术有限公司 | Audio-video synchronizing method and device of set top box |
| CN107509100A (en) * | 2017-09-15 | 2017-12-22 | 深圳国微技术有限公司 | Audio and video synchronization method, system, computer installation and computer-readable recording medium |
| CN107566890A (en) * | 2017-09-15 | 2018-01-09 | 深圳国微技术有限公司 | Handle audio stream broadcasting abnormal method, apparatus, computer installation and computer-readable recording medium |
| CN107801080A (en) * | 2017-11-10 | 2018-03-13 | 普联技术有限公司 | A kind of audio and video synchronization method, device and equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7400653B2 (en) * | 2004-06-18 | 2008-07-15 | Dolby Laboratories Licensing Corporation | Maintaining synchronization of streaming audio and video using internet protocol |
| US20060093320A1 (en) * | 2004-10-29 | 2006-05-04 | Hallberg Bryan S | Operation modes for a personal video recorder using dynamically generated time stamps |
| CN101138246A (en) * | 2005-03-07 | 2008-03-05 | 皇家飞利浦电子股份有限公司 | Buffering of video stream data |
-
2018
- 2018-04-09 CN CN201810310294.6A patent/CN108495164B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11191286A (en) * | 1997-07-28 | 1999-07-13 | Sony Electron Inc | Synchronizing method for audio and video signals and digital video processor |
| CN101827271A (en) * | 2009-03-04 | 2010-09-08 | 联芯科技有限公司 | Audio and video synchronized method and device as well as data receiving terminal |
| CN102075806A (en) * | 2011-01-26 | 2011-05-25 | 四川长虹电器股份有限公司 | Audio and video synchronization method of digital television |
| CN103873952A (en) * | 2012-12-18 | 2014-06-18 | 风网科技（北京）有限公司 | High-efficiency video playing frame loss control system and method |
| CN103607664A (en) * | 2013-11-22 | 2014-02-26 | 天津市北海通信技术有限公司 | Audio and video synchronization method of embedded multimedia playing system |
| CN104202611A (en) * | 2014-09-01 | 2014-12-10 | 赛特斯信息科技股份有限公司 | Method of realizing detection on broken screen defect in video file based on video decoder |
| CN106612452A (en) * | 2015-10-22 | 2017-05-03 | 深圳市中兴微电子技术有限公司 | Audio-video synchronizing method and device of set top box |
| CN107509100A (en) * | 2017-09-15 | 2017-12-22 | 深圳国微技术有限公司 | Audio and video synchronization method, system, computer installation and computer-readable recording medium |
| CN107566890A (en) * | 2017-09-15 | 2018-01-09 | 深圳国微技术有限公司 | Handle audio stream broadcasting abnormal method, apparatus, computer installation and computer-readable recording medium |
| CN107801080A (en) * | 2017-11-10 | 2018-03-13 | 普联技术有限公司 | A kind of audio and video synchronization method, device and equipment |
Non-Patent Citations (2)
| Title |
|---|
| 《基于AVS的嵌入式音视频同步传输系统设计》;郭尧;《中国优秀硕士学位论文全文数据库》;20090415;全文 * |
| 《流媒体系统中音视频同步机制的设计与实现》;杨蓓;《中国优秀硕博士学位论文全文数据库》;20080515;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108495164A (en) | 2018-09-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108495164B (en) | Audio and video synchronization processing method and device, computer device and storage medium | |
| US9794605B2 (en) | Using time-stamped event entries to facilitate synchronizing data streams | |
| US11217279B2 (en) | Method and device for adjusting video playback speed | |
| CN110267083B (en) | Audio and video synchronization detection method, device, equipment and storage medium | |
| US20170083262A1 (en) | System and method for controlling memory frequency using feed-forward compression statistics | |
| CN111277895A (en) | Video frame interpolation method and device | |
| CN108156500B (en) | Multimedia data time correction method, computer device and computer readable storage medium | |
| CN111225171A (en) | Video recording method, device, terminal equipment and computer storage medium | |
| KR100490403B1 (en) | Method for controlling buffering of audio stream and apparatus thereof | |
| CN104113778A (en) | Video stream decoding method and device | |
| CN113923472B (en) | Video content analysis method, device, electronic equipment and storage medium | |
| WO2020035036A1 (en) | Program recording and playback method and device | |
| CN111949512A (en) | Application program jamming detection method and device, terminal and medium | |
| WO2023083213A1 (en) | Data decoding method and apparatus, electronic device and readable storage medium | |
| US9456180B2 (en) | Image processing apparatus, communication system, and computer program | |
| EP1388944A1 (en) | Cross interleave reed-solomon code correction | |
| CN110855645B (en) | Streaming media data playing method and device | |
| CN116233411A (en) | Method, device, equipment and computer storage medium for audio and video synchronous test | |
| US11463493B2 (en) | Method and apparatus for playing media file | |
| CN108335706B (en) | Seamless playing method of multimedia file, terminal equipment and storage medium | |
| US11457287B2 (en) | Method and system for processing video | |
| CN110636380A (en) | Set top box and method for detecting video playing abnormity and abnormity detection system | |
| CN116437161B (en) | Video data processing method, injection method, system, equipment and storage medium | |
| US8699338B2 (en) | Signal processing arrangement and method with adaptable signal reproduction rate | |
| US10911819B2 (en) | Multimedia pipeline device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |