CN111355975B - Live client audio and video delay adjustment playing system and method - Google Patents
Live client audio and video delay adjustment playing system and method Download PDFInfo
- Publication number
- CN111355975B CN111355975B CN202010224624.7A CN202010224624A CN111355975B CN 111355975 B CN111355975 B CN 111355975B CN 202010224624 A CN202010224624 A CN 202010224624A CN 111355975 B CN111355975 B CN 111355975B
- Authority
- CN
- China
- Prior art keywords
- audio
- delay
- video
- data
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
- H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Abstract
The invention provides a live broadcast client audio and video delay adjustment playing system and method. According to the technical scheme, when audio data exists, a frame loss acceleration strategy corresponding to the level is adopted according to the delay level of audio cache delay, and when no audio data exists, a speed doubling acceleration strategy corresponding to the level is adopted according to the delay level of video cache delay. Different acceleration strategies are adopted for different delay levels, the larger the delay is, the more the acceleration is, the gradually smaller the delay is, the acceleration is also gradually reduced, the gradual transition is realized, and the delay is smoothly reduced. Compared with the existing audio and video live broadcast low-delay technical scheme, the method can improve the live broadcast fluency and improve the user experience while ensuring low delay.
Description
Technical Field
The invention relates to the technical field of audio and video live broadcast, in particular to a live broadcast client audio and video delay adjustment playing system and method.
Background
With the development of audio and video real-time communication technology, audio and video live broadcast is widely applied in the fields of education, social contact, finance, entertainment and the like. In most application scenes of audio and video live broadcasting, "low latency" is an important problem to be concerned. In particular, under the condition of abnormal network jitter, a large amount of audio and video data can be accumulated at a client due to a buffering strategy, so that a large delay is generated, and the live broadcast effect is seriously influenced.
In the delay solution in the prior art, generally, when a client plays, a timestamp of audio data is used as a reference, when delay occurs, a frame loss and frame skipping manner is directly adopted for the audio data, and a video correspondingly loses frames or loses packets, so that delay is reduced. According to the scheme, low delay is guaranteed, meanwhile, the playing quality of the audio and video is often greatly influenced, the situations of video screen splash, audio interruption and noise occur, and the problem of video blocking can be caused when the audio playing is in a problem or no audio data exists.
Disclosure of Invention
The invention aims to provide a live broadcast client audio and video delay adjusting and playing system and a live broadcast client audio and video delay adjusting and playing method aiming at the delay caused by poor network state of a client during audio and video live broadcast.
In order to achieve the above object, the technical solution of the present invention provides a live broadcast client audio/video delay adjustment playing system, which includes an audio data cache module, a video data cache module, a delay adjustment module, an audio playing module, and a video playing module; the delay adjusting module is respectively connected with the audio data caching module, the video data caching module, the audio playing module and the video playing module; the audio data cache module is connected with the audio playing module, and the video data cache module is connected with the video playing module;
the audio data caching module comprises an audio caching unit; the audio data caching module decodes the received audio data and judges whether the time stamp of the audio data is larger than the current synchronous time stamp, if so, the audio data is put into an audio caching unit, otherwise, the audio data is discarded;
the video data caching module comprises a video caching unit; the video data caching module is used for placing the received video data into the video caching unit; when data exist in the audio cache unit, the delay adjusting module calculates the current audio cache delay according to the difference of audio data timestamps in the audio cache unit, then compares the audio cache delay with a preset audio multistage delay segmentation threshold value, judges the delay level of the audio cache delay, and updates the current synchronization timestamp according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay;
when no data exists in the audio cache unit, the delay adjusting module calculates the current video cache delay according to the difference of timestamps of the data in the video cache unit, compares the video cache delay with a preset video multistage delay segmentation threshold value, judges the delay level of the video cache delay, and updates the current synchronization timestamp according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay;
the audio playing module reads audio data corresponding to the time stamp and the current synchronous time stamp from the audio cache unit according to the current synchronous time stamp and plays the audio data;
and the video playing module reads the video data with the timestamp corresponding to the current synchronization timestamp and/or before the current synchronization timestamp from the video cache unit according to the current synchronization timestamp, and decodes and plays the video data.
Based on the above, the updating the current synchronization timestamp according to the acceleration policy of dropping one frame every preset number of frames corresponding to the delay level of the audio cache delay includes: according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay, if the first frame data in the audio cache unit needs to be discarded, the updated current synchronization timestamp is equal to the timestamp of the second frame data in the audio cache unit, otherwise, the updated current synchronization timestamp is equal to the timestamp of the first frame data in the audio cache unit.
Based on the above, the updating the current synchronization timestamp according to the preset video speed doubling policy corresponding to the delay level of the video cache delay includes: and according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay, the updated current synchronization timestamp is equal to the current synchronization timestamp before updating plus the product of the time interval between the current updating and the last updating and the speed doubling value.
Based on the above, when there is data in the audio buffer unit, the delay adjustment module updates the current synchronization timestamp according to the update of the current audio playing timestamp; and when no data exists in the audio cache unit, the delay adjusting module updates the current synchronous timestamp according to the update of the current video playing timestamp.
Based on the above, the larger the audio buffer delay is, the smaller the number of frames with the preset number separated by frame loss in the acceleration strategy corresponding to the corresponding delay level is; the larger the video cache delay is, the higher the preset video speed multiplier corresponding to the corresponding delay level is.
Based on the above, the audio data caching module decodes the received audio data, and determines whether the timestamp of the audio data is greater than the current synchronization timestamp, or is less than the current synchronization timestamp but the difference is less than a preset time difference threshold, if so, the audio data is put into an audio caching unit, otherwise, the audio data is discarded.
The technical scheme of the invention also provides a live client audio and video delay adjustment playing method, which comprises the following steps:
s1, decoding the received audio data, and judging whether the time stamp of the audio data is larger than the current synchronous time stamp, if so, putting the audio data into an audio buffer unit, otherwise, discarding the audio data; putting the received video data into a video cache unit;
s2, when data exist in the audio cache unit, calculating the current audio cache delay according to the difference of audio data timestamps in the audio cache unit, then comparing the audio cache delay with a preset audio multistage delay segmentation threshold, judging the delay level of the audio cache delay, and updating the current synchronization timestamp according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay;
when no data exists in the audio cache unit, calculating the current video cache delay according to the difference of timestamps of the data in the video cache unit, then comparing the video cache delay with a preset video multi-level delay segmentation threshold value, judging the delay level of the video cache delay, and updating the current synchronization timestamp according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay;
s3, according to the current synchronization timestamp, reading audio data corresponding to the timestamp and the current synchronization timestamp from the audio cache unit, and playing the audio data; reading video data with a timestamp corresponding to and/or preceding the current synchronization timestamp from the video cache unit, and decoding and playing the video data;
s4, updating the current audio playing time stamp and the current video playing time stamp, and then performing S1.
Based on the above, the updating the current synchronization timestamp according to the acceleration policy of dropping one frame every preset number of frames corresponding to the delay level of the audio buffering delay in step S2 includes: according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay, if the first frame data in the audio cache unit needs to be discarded, the updated current synchronization timestamp is equal to the timestamp of the second frame data in the audio cache unit, otherwise, the updated current synchronization timestamp is equal to the timestamp of the first frame data in the audio cache unit.
Based on the above, the updating the current synchronization timestamp according to the preset video speed doubling policy corresponding to the delay level of the video cache delay includes: and according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay, the updated current synchronization timestamp is equal to the current synchronization timestamp before updating plus the product of the time interval between the current updating and the last updating and the speed doubling value.
Based on the above, the larger the audio buffer delay is, the smaller the number of frames with the preset number separated by frame loss in the acceleration strategy corresponding to the corresponding delay level is; the larger the video cache delay is, the higher the preset video speed multiplier corresponding to the corresponding delay level is.
According to the technical scheme, when audio data exist, a frame loss acceleration strategy corresponding to the level is adopted according to the delay level of audio cache delay, and when no audio data exist, a speed doubling acceleration strategy corresponding to the level is adopted according to the delay level of video cache delay, so that delay is smoothly reduced in a gradual transition mode, and smoothness and real-time performance of audio and video live broadcast are improved. Compared with the scheme in the prior art, the technical scheme of the invention can smoothly transit in the live broadcasting process to reduce the delay and improve the user experience.
Compared with the prior art, the invention has prominent substantive characteristics and remarkable progress, in particular:
1. the technical scheme of the invention determines the delay level according to the audio and video cache delay, different levels adopt different acceleration strategies, the larger the delay is, the more the acceleration is, the gradually smaller the delay is, the acceleration is also gradually reduced, the gradual transition is realized, and the delay is smoothly reduced.
2. According to the technical scheme, when audio data exist, audio and video delay is gradually and synchronously reduced by taking the audio timestamp as a reference, and when audio data do not exist, video delay is gradually reduced by taking the video timestamp as a reference.
Drawings
Fig. 1 is a schematic structural diagram of a live client audio/video delay adjustment playing system according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a live client audio/video delay adjustment playing method according to another embodiment of the present invention.
Detailed Description
Example 1
The embodiment provides an audio and video delay adjustment playing system for a live client, as shown in fig. 1, the system includes an audio data cache module, a video data cache module, a delay adjustment module, an audio playing module, and a video playing module; the delay adjusting module is respectively connected with the audio data caching module, the video data caching module, the audio playing module and the video playing module; the audio data cache module is connected with the audio playing module, and the video data cache module is connected with the video playing module;
the audio data caching module comprises an audio caching unit; the audio data caching module decodes the received audio data and judges whether the time stamp of the audio data is larger than the current synchronous time stamp, if so, the audio data is put into an audio caching unit, otherwise, the audio data is discarded;
the video data caching module comprises a video caching unit; the video data caching module is used for placing the received video data into the video caching unit;
when data exist in the audio cache unit, the delay adjusting module calculates the current audio cache delay according to the difference of audio data timestamps in the audio cache unit, then compares the audio cache delay with a preset audio multistage delay segmentation threshold value, judges the delay level of the audio cache delay, and updates the current synchronization timestamp according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay;
when no data exists in the audio cache unit, the delay adjusting module calculates the current video cache delay according to the difference of timestamps of the data in the video cache unit, compares the video cache delay with a preset video multistage delay segmentation threshold value, judges the delay level of the video cache delay, and updates the current synchronization timestamp according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay;
the audio playing module reads audio data corresponding to the time stamp and the current synchronous time stamp from the audio cache unit according to the current synchronous time stamp and plays the audio data;
and the video playing module reads the video data with the timestamp corresponding to the current synchronization timestamp and/or before the current synchronization timestamp from the video cache unit according to the current synchronization timestamp, and decodes and plays the video data.
The live broadcast client audio and video delay adjusting and playing system in the technical scheme of the invention can be used in combination with various audio and video real-time communication systems, platforms and tools, so that the delay is reduced and the user experience is improved.
And the synchronous time stamp is used, so that the synchronous playing of the audio and video can be ensured. The synchronization time stamp is preferably based on the time stamp of the audio data, and is based on the video time stamp when there is no audio data.
When the audio buffer unit has data, the delay adjusting module calculates the current audio buffer delay according to the difference between the timestamps of the first frame data and the last frame data in the audio buffer unit, wherein the first frame and the last frame respectively refer to the frames with the minimum and the maximum timestamps of the audio data.
When the audio buffer unit has no data, the delay adjusting module calculates the current video buffer delay according to the difference between the timestamps of the first frame data and the last frame data in the video buffer unit, wherein the first frame and the last frame respectively refer to the frames with the minimum and the maximum timestamps of the video data.
And judging the delay level of the audio cache delay according to the comparison between the audio cache delay and the segmented threshold, wherein different delay levels correspond to different acceleration strategies. In this embodiment, the following example of an acceleration strategy is given:
when the audio cache duration is more than 16s, corresponding to the delay level 1, the acceleration strategy is to lose one frame every other frame;
when the audio cache time length is more than 12s and less than or equal to 16s, corresponding to the delay level 2, the acceleration strategy is to lose one frame every 3 frames;
when the audio cache time is more than 8s and less than or equal to 12s, corresponding to the delay level 3, the acceleration strategy is to lose one frame every 4 frames;
when the audio cache time is more than 5s and less than or equal to 8s, corresponding to the delay level 4, the acceleration strategy is to lose one frame every 5 frames;
when the audio cache time is more than 3s and less than or equal to 5s, corresponding to the delay level 5, the acceleration strategy is to lose one frame every 6 frames;
when the audio cache time length is more than 1s and less than or equal to 3s, corresponding to the delay level 6, the acceleration strategy is to lose one frame every 10 frames;
when the audio buffering time is less than or equal to 1s, corresponding to the delay level 7, the frame is not lost in normal playing.
And judging the delay level of the video cache delay according to the comparison between the video cache delay and the segmented threshold, wherein different delay levels correspond to different acceleration strategies. In this embodiment, the following example of an acceleration strategy is given:
when the video cache duration is more than 16s, corresponding to the delay level 1, the acceleration strategy is 2 times speed playing;
when the video caching time is more than 12s and less than or equal to 16s, corresponding to the delay level 2, the acceleration strategy is played at a speed of 1.8 times;
when the video cache time is more than 8s and less than or equal to 12s, corresponding to the delay level 3, the acceleration strategy is played at the speed of 1.6 times;
when the video cache time is more than 5s and less than or equal to 8s, corresponding to the delay level 4, the acceleration strategy is 1.4 times speed playing;
when the video cache time is more than 3s and less than or equal to 5s, corresponding to the delay level 5, the acceleration strategy is 1.2 times speed playing;
when the video cache time is more than 1s and less than or equal to 3s, corresponding to the delay level 6, the acceleration strategy is 1.1 times speed playing;
and when the video caching time is less than or equal to 1s, normally playing the video corresponding to the delay level 7.
In this embodiment, the updating the current synchronization timestamp according to the acceleration policy of every preset number of frames corresponding to the delay level of the audio cache delay includes: according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay, if the first frame data in the audio cache unit needs to be discarded, the updated current synchronization timestamp is equal to the timestamp of the second frame data in the audio cache unit, otherwise, the updated current synchronization timestamp is equal to the timestamp of the first frame data in the audio cache unit.
In this embodiment, the updating the current synchronization timestamp according to the preset video speed doubling policy corresponding to the delay level of the video caching delay includes: and according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay, the updated current synchronization timestamp is equal to the current synchronization timestamp before updating plus the product of the time interval between the current updating and the last updating and the speed doubling value.
In this embodiment, when there is data in the audio buffer unit, the delay adjustment module updates the current synchronization timestamp according to the update of the current audio playing timestamp; and when no data exists in the audio cache unit, the delay adjusting module updates the current synchronous timestamp according to the update of the current video playing timestamp.
The current audio playing time stamp and the current video playing time stamp are updated to the time stamp of the currently played audio/video data or to the current synchronization time stamp along with the playing of the audio and video, respectively.
In this embodiment, the greater the audio buffering delay, the smaller the number of frames of the preset number separated by frame loss in the acceleration strategy corresponding to the corresponding delay level; the larger the video cache delay is, the higher the preset video speed multiplier corresponding to the corresponding delay level is.
Example 2
The difference between this embodiment and embodiment 1 is that, in this embodiment, the audio data caching module decodes the received audio data, and determines whether a timestamp of the audio data is greater than the current synchronization timestamp, or is less than the current synchronization timestamp but the difference is less than a preset time difference threshold, if so, the audio data is put into an audio caching unit, otherwise, the audio data is discarded.
For example, a time difference threshold of 100ms may be set, which leaves a 100ms audio data compatibility mechanism.
The part of the audio data which is smaller than the current synchronization timestamp but has a difference smaller than a preset time difference threshold in the audio buffer unit can be used for audio data backtracking. This portion of audio data is also updated as the current synchronization timestamp changes.
In this embodiment, when there is data in the audio cache unit, the delay adjustment module calculates the current audio cache delay according to a difference between timestamps of the first frame data and the last frame data of which timestamps are greater than the current synchronization timestamp in the audio cache unit.
Example 3
Based on the same inventive concept as the above method, another embodiment of the present invention further provides a live broadcast client audio/video delay adjustment playing method, as shown in fig. 2, the method includes the following steps:
s1, decoding the received audio data, and judging whether the time stamp of the audio data is larger than the current synchronous time stamp, if so, putting the audio data into an audio buffer unit, otherwise, discarding the audio data; putting the received video data into a video cache unit;
s2, when data exist in the audio cache unit, calculating the current audio cache delay according to the difference of audio data timestamps in the audio cache unit, then comparing the audio cache delay with a preset audio multistage delay segmentation threshold, judging the delay level of the audio cache delay, and updating the current synchronization timestamp according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay;
when no data exists in the audio cache unit, calculating the current video cache delay according to the difference of timestamps of the data in the video cache unit, then comparing the video cache delay with a preset video multi-level delay segmentation threshold value, judging the delay level of the video cache delay, and updating the current synchronization timestamp according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay;
s3, according to the current synchronization timestamp, reading audio data corresponding to the timestamp and the current synchronization timestamp from the audio cache unit, and playing the audio data; reading video data with a timestamp corresponding to and/or preceding the current synchronization timestamp from the video cache unit, and decoding and playing the video data;
s4, updating the current audio playing time stamp and the current video playing time stamp, and then performing S1.
In this embodiment, the updating the current synchronization timestamp according to the acceleration policy of dropping one frame every preset number of frames corresponding to the delay level of the audio buffering delay in step S2 includes: according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay, if the first frame data in the audio cache unit needs to be discarded, the updated current synchronization timestamp is equal to the timestamp of the second frame data in the audio cache unit, otherwise, the updated current synchronization timestamp is equal to the timestamp of the first frame data in the audio cache unit.
In this embodiment, the updating the current synchronization timestamp according to the preset video speed doubling policy corresponding to the delay level of the video caching delay includes: and according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay, the updated current synchronization timestamp is equal to the current synchronization timestamp before updating plus the product of the time interval between the current updating and the last updating and the speed doubling value.
In this embodiment, the greater the audio buffering delay, the smaller the number of frames of the preset number separated by frame loss in the acceleration strategy corresponding to the corresponding delay level; the larger the video cache delay is, the higher the preset video speed multiplier corresponding to the corresponding delay level is.
The above step numbers do not represent a sequential order and those skilled in the art can change the order without departing from the scope of the present invention. The above-mentioned contents are only for illustrating the technical idea of the present invention, and the protection scope of the present invention is not limited thereby, and any modification made on the basis of the technical idea of the present invention falls within the protection scope of the claims of the present invention.
Claims (10)
1. A live broadcast customer end audio frequency and video time delay adjusts broadcast system which characterized in that: the device comprises an audio data caching module, a video data caching module, a delay adjusting module, an audio playing module and a video playing module; the delay adjusting module is respectively connected with the audio data caching module, the video data caching module, the audio playing module and the video playing module; the audio data cache module is connected with the audio playing module, and the video data cache module is connected with the video playing module;
the audio data caching module comprises an audio caching unit; the audio data caching module decodes the received audio data and judges whether the time stamp of the audio data is larger than the current synchronous time stamp, if so, the audio data is put into an audio caching unit, otherwise, the audio data is discarded;
the video data caching module comprises a video caching unit; the video data caching module is used for placing the received video data into the video caching unit;
when data exist in the audio cache unit, the delay adjusting module calculates the current audio cache delay according to the difference of audio data timestamps in the audio cache unit, then compares the audio cache delay with a preset audio multistage delay segmentation threshold value, judges the delay level of the audio cache delay, and updates the current synchronization timestamp according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay;
when the audio cache unit has data, the delay adjusting module calculates the current audio cache delay according to the difference of the timestamps of the first frame data and the last frame data in the audio cache unit, wherein the first frame and the last frame respectively refer to the frames with the smallest and the largest timestamps of the audio data;
when no data exists in the audio cache unit, the delay adjusting module calculates the current video cache delay according to the difference of timestamps of the data in the video cache unit, compares the video cache delay with a preset video multistage delay segmentation threshold value, judges the delay level of the video cache delay, and updates the current synchronization timestamp according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay;
when no data exists in the audio cache unit, the delay adjusting module calculates the current video cache delay according to the difference of timestamps of the first frame data and the last frame data in the video cache unit, wherein the first frame and the last frame respectively refer to the frames with the minimum and the maximum timestamps of the video data;
the audio playing module reads audio data corresponding to the time stamp and the current synchronous time stamp from the audio cache unit according to the current synchronous time stamp and plays the audio data;
and the video playing module reads the video data with the timestamp corresponding to the current synchronization timestamp and/or before the current synchronization timestamp from the video cache unit according to the current synchronization timestamp, and decodes and plays the video data.
2. The live client audio-video delay adjustment playing system according to claim 1, characterized in that: the updating the current synchronization timestamp according to the acceleration policy of discarding one frame every preset number of frames corresponding to the delay level of the audio cache delay includes: according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay, if the first frame data in the audio cache unit needs to be discarded, the updated current synchronization timestamp is equal to the timestamp of the second frame data in the audio cache unit, otherwise, the updated current synchronization timestamp is equal to the timestamp of the first frame data in the audio cache unit.
3. The live client audio-video delay adjustment playing system according to claim 1, characterized in that: the updating the current synchronization timestamp according to the preset video speed doubling strategy corresponding to the delay level of the video cache delay comprises: and according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay, the updated current synchronization timestamp is equal to the current synchronization timestamp before updating plus the product of the time interval between the current updating and the last updating and the speed doubling value.
4. The live client audio-video delay adjustment playing system according to claim 1, characterized in that: when the audio cache unit has data, the delay adjusting module updates the current synchronous timestamp according to the update of the current audio playing timestamp; and when no data exists in the audio cache unit, the delay adjusting module updates the current synchronous timestamp according to the update of the current video playing timestamp.
5. The live client audio-video delay adjustment playing system according to claim 1, characterized in that: the larger the audio buffer delay is, the smaller the number of frames with preset number separated by frame loss in an acceleration strategy corresponding to the corresponding delay level is; the larger the video cache delay is, the higher the preset video speed multiplier corresponding to the corresponding delay level is.
6. The live client audio-video delay adjustment playing system according to claim 1, characterized in that: the audio data caching module decodes the received audio data, judges whether the time stamp of the audio data is larger than the current synchronization time stamp or smaller than the current synchronization time stamp but the difference value is smaller than a preset time difference threshold value, puts the audio data into an audio caching unit if the time stamp of the audio data is smaller than the current synchronization time stamp, and discards the audio data if the time stamp of the audio data is smaller than the current synchronization time stamp of the audio data but the difference value is smaller than the preset time difference threshold value.
7. A live broadcast client audio and video delay adjustment playing method is characterized by comprising the following steps:
s1, decoding the received audio data, and judging whether the time stamp of the audio data is larger than the current synchronous time stamp, if so, putting the audio data into an audio buffer unit, otherwise, discarding the audio data; putting the received video data into a video cache unit;
s2, when data exist in the audio cache unit, calculating the current audio cache delay according to the difference of audio data timestamps in the audio cache unit, then comparing the audio cache delay with a preset audio multistage delay segmentation threshold, judging the delay level of the audio cache delay, and updating the current synchronization timestamp according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay;
when the audio cache unit has data, the delay adjusting module calculates the current audio cache delay according to the difference of the timestamps of the first frame data and the last frame data in the audio cache unit, wherein the first frame and the last frame respectively refer to the frames with the smallest and the largest timestamps of the audio data;
when no data exists in the audio cache unit, calculating the current video cache delay according to the difference of timestamps of the data in the video cache unit, then comparing the video cache delay with a preset video multi-level delay segmentation threshold value, judging the delay level of the video cache delay, and updating the current synchronization timestamp according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay;
when no data exists in the audio cache unit, the delay adjusting module calculates the current video cache delay according to the difference of timestamps of the first frame data and the last frame data in the video cache unit, wherein the first frame and the last frame respectively refer to the frames with the minimum and the maximum timestamps of the video data;
s3, according to the current synchronization timestamp, reading audio data corresponding to the timestamp and the current synchronization timestamp from the audio cache unit, and playing the audio data; reading video data with a timestamp corresponding to and/or preceding the current synchronization timestamp from the video cache unit, and decoding and playing the video data;
s4, updating the current audio playing time stamp and the current video playing time stamp, and then performing S1.
8. The live client audio-video delay adjustment playing method according to claim 7, characterized in that: the updating the current synchronization timestamp according to the acceleration policy of discarding one frame every preset number of frames corresponding to the delay level of the audio buffering delay in step S2 includes: according to an acceleration strategy that one frame is discarded every other preset number of frames corresponding to the delay level of the audio cache delay, if the first frame data in the audio cache unit needs to be discarded, the updated current synchronization timestamp is equal to the timestamp of the second frame data in the audio cache unit, otherwise, the updated current synchronization timestamp is equal to the timestamp of the first frame data in the audio cache unit.
9. The live client audio-video delay adjustment playing method according to claim 7, characterized in that: the updating the current synchronization timestamp according to the preset video speed doubling strategy corresponding to the delay level of the video cache delay comprises: and according to a preset video speed doubling strategy corresponding to the delay level of the video cache delay, the updated current synchronization timestamp is equal to the current synchronization timestamp before updating plus the product of the time interval between the current updating and the last updating and the speed doubling value.
10. The live client audio-video delay adjustment playing method according to claim 7, characterized in that: the larger the audio buffer delay is, the smaller the number of frames with preset number separated by frame loss in an acceleration strategy corresponding to the corresponding delay level is; the larger the video cache delay is, the higher the preset video speed multiplier corresponding to the corresponding delay level is.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010224624.7A CN111355975B (en) | 2020-03-26 | 2020-03-26 | Live client audio and video delay adjustment playing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010224624.7A CN111355975B (en) | 2020-03-26 | 2020-03-26 | Live client audio and video delay adjustment playing system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111355975A CN111355975A (en) | 2020-06-30 |
| CN111355975B true CN111355975B (en) | 2021-09-07 |
Family
ID=71196371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010224624.7A Active CN111355975B (en) | 2020-03-26 | 2020-03-26 | Live client audio and video delay adjustment playing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111355975B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111885334A (en) * | 2020-08-26 | 2020-11-03 | 杭州速递科技有限公司 | Method for reducing delay of real-time frame pursuit of audio and video |
| CN113490055B (en) * | 2021-07-06 | 2023-09-19 | 三星电子（中国）研发中心 | Data processing method and device |
| CN113707174B (en) * | 2021-08-31 | 2024-02-09 | 亿览在线网络技术（北京）有限公司 | Method for generating animation special effects driven by audio |
| CN114285513B (en) * | 2021-11-22 | 2023-10-27 | 杭州当虹科技股份有限公司 | Delay device and method for supporting long-time delay of lossless IP signal |
| CN114363710A (en) * | 2022-01-04 | 2022-04-15 | 上海哔哩哔哩科技有限公司 | Live broadcast watching method and device based on time shifting acceleration |
| CN114827679A (en) * | 2022-04-19 | 2022-07-29 | 海信视像科技股份有限公司 | Display device and sound picture synchronization method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103581730A (en) * | 2013-10-28 | 2014-02-12 | 南京熊猫电子股份有限公司 | Method for achieving synchronization of audio and video on digital set top box |
| CN104092920A (en) * | 2014-07-16 | 2014-10-08 | 浙江航天长峰科技发展有限公司 | Audio and video synchronizing method |
| CN104113777A (en) * | 2014-08-01 | 2014-10-22 | 广州金山网络科技有限公司 | Audio stream decoding method and device |
| CN104782109A (en) * | 2012-10-30 | 2015-07-15 | 三菱电机株式会社 | Audio/video play system, video display device, and audio output device |
| CN105898507A (en) * | 2016-05-30 | 2016-08-24 | 广东威创视讯科技股份有限公司 | Video signal synchronization method and device |
| CN106603543A (en) * | 2016-12-22 | 2017-04-26 | 努比亚技术有限公司 | Corrected streaming media audio and video synchronizing method and apparatus |
| CN110248204A (en) * | 2019-07-16 | 2019-09-17 | 广州虎牙科技有限公司 | A kind of processing method, device, equipment and the storage medium of live streaming caching |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9531921B2 (en) * | 2013-08-30 | 2016-12-27 | Audionow Ip Holdings, Llc | System and method for video and secondary audio source synchronization |
| US9898077B2 (en) * | 2013-09-18 | 2018-02-20 | Booktrack Holdings Limited | Playback system for synchronised soundtracks for electronic media content |
-
2020
- 2020-03-26 CN CN202010224624.7A patent/CN111355975B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104782109A (en) * | 2012-10-30 | 2015-07-15 | 三菱电机株式会社 | Audio/video play system, video display device, and audio output device |
| CN103581730A (en) * | 2013-10-28 | 2014-02-12 | 南京熊猫电子股份有限公司 | Method for achieving synchronization of audio and video on digital set top box |
| CN104092920A (en) * | 2014-07-16 | 2014-10-08 | 浙江航天长峰科技发展有限公司 | Audio and video synchronizing method |
| CN104113777A (en) * | 2014-08-01 | 2014-10-22 | 广州金山网络科技有限公司 | Audio stream decoding method and device |
| CN105898507A (en) * | 2016-05-30 | 2016-08-24 | 广东威创视讯科技股份有限公司 | Video signal synchronization method and device |
| CN106603543A (en) * | 2016-12-22 | 2017-04-26 | 努比亚技术有限公司 | Corrected streaming media audio and video synchronizing method and apparatus |
| CN110248204A (en) * | 2019-07-16 | 2019-09-17 | 广州虎牙科技有限公司 | A kind of processing method, device, equipment and the storage medium of live streaming caching |
Non-Patent Citations (2)
| Title |
|---|
| "一种多媒体信息交互系统音视频同步方法";郑睿等;《杭州电子科技大学学报》;20140806(第3期);全文 * |
| "基于智能媒体传输协议的异构网络多源内容同步传输技术研究";王成志;《中国优秀硕士学位论文全文数据库-信息科技辑》;20200315(第 03 期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111355975A (en) | 2020-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111355975B (en) | Live client audio and video delay adjustment playing system and method | |
| US8412364B2 (en) | Method and device for sending and playing streaming data | |
| CN109714634B (en) | Decoding synchronization method, device and equipment for live data stream | |
| CN104618786B (en) | Audio and video synchronization method and device | |
| US10097878B2 (en) | Video playback method and control terminal thereof | |
| KR100968928B1 (en) | Apparatus and method for synchronization of audio and video streams | |
| RU2507707C2 (en) | Method and apparatus for controlling video and audio data reproduction | |
| CN107566918B (en) | A kind of low delay under video distribution scene takes the neutrel extraction of root | |
| CN107529097A (en) | A kind of method and device of adaptive regulating video buffer size | |
| US7940805B2 (en) | Receiving device, clock synchronizing method, and computer program | |
| CN109168083A (en) | A kind of Streaming Media real time playing method and device | |
| KR100643270B1 (en) | Client and method for playing video stream | |
| CN109905763B (en) | Method for optimizing audio playing pause of player | |
| CN112333526B (en) | Video buffer adjustment method and device, storage medium and electronic device | |
| CN108259964A (en) | A kind of video playing rate adjusting method and system | |
| CN108540855B (en) | Self-adaptive low-delay streaming media playing method suitable for network live broadcast scene | |
| Li et al. | Arrival process-controlled adaptive media playout with multiple thresholds for video streaming | |
| EP1528724A1 (en) | Information processing device and method, recording medium, and program | |
| US20220103609A1 (en) | Method and apparatus for playing multimedia streaming data | |
| CN109218809B (en) | Streaming media playing method and device | |
| CN112073823B (en) | Frame loss processing method, video playing terminal and computer readable storage medium | |
| US20080025345A1 (en) | Methods and Systems for Buffer Management | |
| CN117729376A (en) | Video playing method, device, equipment and storage medium | |
| CN114584844B (en) | RTP packet loss retransmission method and device and intelligent set top box | |
| EP2077671B1 (en) | Streaming media player and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A live streaming client audio and video delay adjustment playback system and method Effective date of registration: 20230412 Granted publication date: 20210907 Pledgee: China Construction Bank Corporation Zhengzhou Jinshui sub branch Pledgor: ZHENGZHOU XINDA JIEAN INFORMATION TECHNOLOGY Co.,Ltd. Registration number: Y2023980037751 |