CN115842923A - Timestamp correction method and device and storage medium - Google Patents
Timestamp correction method and device and storage medium Download PDFInfo
- Publication number
- CN115842923A CN115842923A CN202211469291.XA CN202211469291A CN115842923A CN 115842923 A CN115842923 A CN 115842923A CN 202211469291 A CN202211469291 A CN 202211469291A CN 115842923 A CN115842923 A CN 115842923A
- Authority
- CN
- China
- Prior art keywords
- frame data
- time
- audio
- video
- correction value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The embodiment of the application discloses a timestamp correction method and device and a storage medium, wherein the timestamp correction method comprises the following steps: acquiring current frame data and previous frame data from a received audio-video data stream, and determining a time interval between the current frame data and the previous frame data; determining whether time jump abnormity exists according to the time interval; acquiring a timestamp correction value under the condition that time jump abnormity exists; and adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until each frame of the audio-video data stream is adjusted to have frame data with abnormal time jump.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a timestamp correction method and apparatus, and a storage medium.
Background
With the development of internet technology, more and more people prefer live broadcast, and commodities are bought and sold through live broadcast, or communication is carried out with other net friends, or some interesting stories are carried out for the net friends.
In the related technology, the situation that an adaptive code rate Streaming media transport protocol (HTTP Live Streaming, HLS) slicing module based on HTTP slices a Live stream exists in a Live broadcasting process, since the HLS slicing module slices the Live stream according to frames, and slices are named according to timestamps, a slice renaming is caused, that is, under the condition that a timestamp of a first slice is the same as that of a second slice obtained later, the second slice covers the first slice, data loss of the first slice is caused, the situations of stumbling and frame skipping occur during Live stream playing are caused, and therefore fluency during Live data playing is reduced.
Disclosure of Invention
In order to solve the foregoing technical problems, embodiments of the present application are expected to provide a timestamp correction method and apparatus, and a storage medium, which can improve fluency when live data is played.
The technical scheme of the application is realized as follows:
the embodiment of the application provides a timestamp correction method, which comprises the following steps:
acquiring current frame data and previous frame data from a received audio-video data stream, and determining a time interval between the current frame data and the previous frame data;
determining whether time jump abnormity exists according to the time interval;
acquiring a timestamp correction value under the condition that time jump abnormity exists;
and adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until the adjusted frame data with abnormal time jump exists in each frame in the audio and video data stream.
An embodiment of the present application provides a timestamp correction apparatus, the apparatus includes:
the acquisition unit is used for acquiring current frame data and previous frame data from the received audio and video data stream; acquiring a timestamp correction value under the condition that time jump abnormity exists;
a determining unit, configured to determine a time interval between the current frame data and the previous frame data; determining whether time jump abnormity exists according to the time interval;
and the adjusting unit is used for adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until the frame data with abnormal time jump exists in each frame in the audio and video data stream after the adjustment is finished.
An embodiment of the present application provides a timestamp correction apparatus, the apparatus includes:
a memory, a processor, and a communication bus, the memory communicating with the processor through the communication bus, the memory storing a time stamp correction program executable by the processor, the time stamp correction method being performed by the processor when the time stamp correction program is executed.
The embodiment of the present application provides a storage medium, on which a computer program is stored, and is applied to a timestamp correction apparatus, wherein the computer program is executed by a processor to implement the timestamp correction method described above.
The embodiment of the application provides a timestamp correction method and device and a storage medium, wherein the timestamp correction method comprises the following steps: acquiring current frame data and previous frame data from a received audio-video data stream, and determining a time interval between the current frame data and the previous frame data; determining whether time jump abnormity exists according to the time interval; acquiring a timestamp correction value under the condition that time jump abnormity exists; and adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until each frame in the audio and video data stream is adjusted to have frame data with abnormal time jump. By adopting the method, the timestamp correction device determines whether time hopping abnormity exists or not through the time interval between the current frame data and the last frame data acquired from the audio-video data stream, and adjusts the timestamp of the current frame data according to the timestamp correction value under the condition that the time hopping abnormity exists, so that the adjusted current frame data is obtained, frame-level slice data with the same timestamp does not exist in the audio-video data stream, the phenomenon of slice renaming is avoided, the conditions of pause and frame skipping during the playing of the live broadcast stream are avoided, and the smoothness of the live broadcast data during the playing is improved.
Drawings
Fig. 1 is a schematic diagram of a timestamp correction process according to an embodiment of the present application;
fig. 2 is a schematic diagram of an exemplary timestamp correction flow provided by an embodiment of the present application;
fig. 3 is a first schematic structural diagram illustrating a timestamp correction apparatus according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a time stamp correction apparatus according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
An embodiment of the present application provides a timestamp correction method, where the timestamp correction method is applied to a timestamp correction device, fig. 1 is a flowchart of the timestamp correction method provided in the embodiment of the present application, and as shown in fig. 1, the timestamp correction method may include:
s101, acquiring current frame data and previous frame data from the received audio and video data stream, and determining a time interval between the current frame data and the previous frame data.
The timestamp correction method provided by the embodiment of the application is suitable for a scene of correcting the time of frame data in audio and video data streams.
In the embodiment of the present application, the time stamp correction apparatus may be implemented in various forms. For example, the timestamp correction apparatus described in the present application may include apparatuses such as a mobile phone, a camera, a tablet computer, a notebook computer, a palm top computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation apparatus, a wearable device, a smart band, a pedometer, and the like, and apparatuses such as a Digital TV, a desktop computer, a server, and the like.
In this embodiment of the present application, the audio and video data stream may be a Real Time Messaging Protocol (RTMP) stream, and the audio and video data stream may also be an HLS type data stream; the audio-video data stream can also be other data streams containing audio data and video data; the specific audio and video data stream may be determined according to actual situations, which is not limited in this embodiment of the present application.
Illustratively, the stream of audiovisual data may be a stream generated on-air. The audio and video data stream can also be a recorded audio and video data stream; the specific audio and video data stream may be determined according to actual situations, which is not limited in the embodiment of the present application.
It should be noted that, a time correction switch (time _ jitter switch) and a timestamp correction module are further arranged in the timestamp correction device, and when the RTMP stream is received, whether the time _ jitter switch is turned on is detected, and when it is determined that the time correction switch is in an on state, the timestamp correction module is entered, and the timestamp correction module is used to start executing the timestamp correction method in the embodiment of the present application.
In the embodiment of the present application, the current frame data is any frame data in the audio-video data stream. The previous frame data is the previous frame audio and video data corresponding to the current frame.
In the embodiment of the application, the current frame data carries time information of the current frame data; the last frame data carries time information of the last frame data; and obtaining the time interval between the current frame data and the previous frame data by using the difference between the time information of the previous frame data and the time information of the current frame data.
In the embodiment of the present application, the previous frame data includes previous frame audio data and previous frame video data; the process of the time stamp correcting apparatus determining the time interval between the current frame data and the previous frame data includes: the time stamp correction device determines an audio frame time interval according to the current audio frame data and the previous frame of audio data; determining a video frame time interval according to current video frame data and previous frame video data; the audio frame time interval and the video frame time interval are taken as time intervals.
It should be noted that the current frame data includes current audio frame data and current video frame data. The time intervals include audio frame time intervals and video frame time intervals.
In the embodiment of the application, the current audio frame data carries the time information of the current audio frame; the previous frame of audio data carries time information corresponding to the previous frame of audio frame; and obtaining the audio frame time interval by using the difference value between the time information of the current audio frame and the time information corresponding to the previous audio frame.
In the embodiment of the application, the current video frame data carries the time information of the current video frame; the previous frame of video data carries time information corresponding to the previous frame of video data; the difference between the time information of the current video frame and the time information corresponding to the previous video frame is used to obtain the video frame time interval.
In the embodiment of the application, before the timestamp correction device acquires the current frame data and the previous frame data from the received audio and video data stream, the timestamp correction device also detects whether the adjacent data frames in the audio and video data stream have time jump abnormity; under the condition that the time jump abnormity exists, marking the state of the first frame data with the time jump abnormity as a time jump abnormity state; and taking the first frame data as the current frame data.
In this embodiment of the present application, the timestamp correction apparatus includes an HLS slicing module, which is configured to slice an RTMP stream to obtain an HLS type data stream when the RTMP stream is received.
In the embodiment of the present application, after receiving the RTMP stream, the HLS slicing module performs a time stamp correction method based on the slicing time stamp (the time stamp of the first frame in the RTMP stream) and corrects only the time stamp of the frame data in the current RTMP stream. If the time stamp jumps in the slicing process, recording a jumping state (the state of marking the first frame data with time jumping abnormity is a time jumping abnormity state), then acquiring a time stamp correction value, and adjusting the time stamp of the current frame data according to the time stamp correction value to obtain the adjusted current frame data; until each frame data after the current frame data in the RTMP stream is adjusted.
In the embodiment of the application, when a slice is newly generated by using an HLS slicing module, if a timestamp jump occurs in a certain slice (frame data), EXT-X-DISCONTINUITY is added before the slice, which indicates that the slice has a timestamp interruption, and the newly generated slice is increased according to the latest timestamp reference after the jump; acquiring current frame data and previous frame data, and determining a time interval between the current frame data and the previous frame data; determining whether time jump abnormity exists according to the time interval; acquiring a timestamp correction value under the condition that time jump abnormity exists; and adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until each frame in the audio and video data stream is adjusted to have frame data with abnormal time jump.
For example, on the basis of taking the current system time as the slice name, if the system time of the current generation slice (alice) is less than or equal to the generation time (file name) of the last slice (B slice), the name of the current generation slice (a slice) is incremented by 1 after the last slice (B slice) generates the name. Therefore, under the condition that a plurality of slices are generated instantly (namely the cached system time is unchanged), the name (system time) of the later generated slice is at least 1 more than the name (system time) of the last slice, and the plurality of slices are not renamed and the phenomenon of audio and video frame loss is avoided. And when the time stamps are compared and the maximum time of the slices is calculated, the same type of frames are adopted for comparison, namely audio is compared with audio and video is compared with video. Therefore, the problem of frame level slicing caused by that the time stamps of the audio and video frame data are monotonically increased but the time stamps of the audio and video frame data are greatly different can be solved. And if the time stamp jumps in the slicing process, recording the jumping state, and adding a time stamp correction value to enable the current slice to be increased progressively according to the time stamp reference before jumping until a new next slice is generated. When a slice is newly generated, if the timestamp jump of the previous slice occurs, EXT-X-DISCONTINUITY is added before the slice, the existence of the timestamp interruption of the slice is identified, and the newly generated slice is increased according to the latest timestamp reference after the jump. And (3) the time stamp returns to zero, the recorded video time stamps are sequentially increased from 0, and after the time stamp jumping occurs, the jumping abnormity is filled by adopting the time stamp correction value. It is guaranteed that here the flow's timestamp on the current slice server can be corrected to a flow starting from 0.
And S102, determining whether time jump abnormity exists according to the time interval.
In this embodiment, after the time stamp correcting apparatus determines the time interval between the current frame data and the previous frame data, the time stamp correcting apparatus may determine whether a time jump anomaly exists according to the time interval.
In this embodiment of the present application, a process of determining whether there is a time jump anomaly by a timestamp correction apparatus according to a time interval includes: the timestamp correction device determines that time jump abnormity exists under the condition that the time interval is a negative value; or determining that the time jump abnormity exists under the condition that the time interval is greater than or equal to the preset time interval.
In the embodiment of the present application, the preset time interval may be a time interval configured in the timestamp correction apparatus; the preset time interval can also be a time interval transmitted to the timestamp correction device by other equipment; the specific way in which the timestamp correction device obtains the preset time interval may be determined according to actual conditions, which is not limited in the embodiment of the present application.
It should be noted that the preset time interval may be 250 milliseconds, and the preset time interval may also be 200 milliseconds; the preset time interval may also be other time interval values; the specific preset time interval is determined according to actual conditions, and the embodiment of the application is not limited to this.
In the embodiment of the present application, in the case that the time interval is a negative value, that is, the time of the current frame data is earlier than the time of the previous frame data. Normally, the time of the previous frame data needs to be earlier than the time of the current frame data, so that in the case that the time of the current frame data is earlier than the time of the previous frame data, the time interval can be considered to determine whether the time jump abnormality exists.
In this embodiment of the present application, the time interval includes an audio frame time interval and a video frame time interval, and the process of determining that there is a time jump anomaly by the timestamp correction apparatus when the time interval is a negative value includes: under the condition that the time interval of the audio frame is a negative value, determining that time jump abnormity exists in the audio frame data; or determining that the video frame data has time jump abnormity under the condition that the video frame time interval is a negative value.
In this embodiment of the application, the timestamp correcting apparatus may also obtain at least one second historical time interval between two adjacent frames of audio frame data before the current frame data from the obtained audio-video data stream, determine a first average time interval of the audio frame data according to the at least one second historical time interval, and determine that the audio frame data has a time jump abnormality if an absolute value of a difference between the audio frame time interval and the first average time interval is greater than or equal to a first preset difference.
In this embodiment of the application, the timestamp correcting apparatus may also obtain, from the obtained audio-visual data stream, at least one first historical time interval between two adjacent frames of video frame data before the current frame data, determine a second average time interval of the video frame data according to the at least one first historical time interval, and determine that there is a time jump abnormality in the video frame data when an absolute value of a difference between the video frame time interval and the second average time interval is greater than or equal to a second preset difference.
It should be noted that the preset difference (the first preset difference and the second preset difference) is a difference configured in the timestamp correction apparatus; the preset difference value can also be a difference value transmitted to the timestamp correction device by other equipment; the specific way in which the timestamp correction device obtains the preset difference value may be determined according to actual conditions, which is not limited in the embodiment of the present application.
It should be further noted that the first preset difference and the second preset difference may be the same, or the first preset difference and the second preset difference may be different; the specific method can be determined according to actual conditions, and the embodiment of the present application does not limit this.
In this embodiment of the present application, when the time interval is greater than or equal to the preset time interval, the timestamp correction apparatus determines that a time jump anomaly exists, including: determining that time jump abnormity exists in the audio frame data under the condition that the audio frame time interval is greater than or equal to a preset audio frame time interval; or determining that the video frame data has time jump abnormity under the condition that the video frame time interval is greater than or equal to the preset video frame time interval.
In the embodiment of the present application, the preset time interval includes a preset audio frame time interval and a preset video frame time interval.
And S103, acquiring a timestamp correction value under the condition that time jump abnormity exists.
In the embodiment of the present application, after determining whether a time jump anomaly exists according to a time interval, the timestamp correction device acquires the timestamp correction value when determining that the time jump anomaly exists.
In the embodiment of the present application, the timestamp correction value may be a configured correction value; the timestamp correction value can also be a value transmitted to the timestamp correction device by other equipment; the time stamp correction value may also be a value determined by the time stamp correction apparatus in a preset manner; the specific way for the timestamp correction device to obtain the timestamp correction value may be determined according to actual conditions, which is not limited in the embodiment of the present application.
Illustratively, the timestamp correction value may be an empirical value configured by the operation and maintenance personnel.
In the embodiment of the application, before the timestamp correction device acquires the timestamp correction value, the timestamp correction device also acquires a first historical time interval between two adjacent frames of video frame data before the current frame data in the recorded audio and video data stream; determining a video temporal correction value according to the first historical time interval; determining the sampling rate of audio frame data according to the audio and video data stream; determining an audio time correction value according to the sampling rate; the video time correction value and the audio time correction value are taken as time stamp correction values.
It should be noted that the number of the first historical time intervals may be one; the number of the first historical time intervals can also be multiple; i.e. the number of first historical time intervals is at least one.
In this embodiment of the application, the timestamp correction apparatus may obtain the video time correction value by determining the video time correction value according to at least one first historical time interval, which is an average time interval of the at least one first historical time interval.
In the embodiment of the present application, the timestamp correction apparatus determines the audio time correction value according to the sampling rate, as shown in formula (1):
wherein 1000 refers to time 1000 milliseconds, i.e. 1 second; 1024 is the number of points sampled for 1 second.
Note that the time stamp correction value includes a video time correction value and an audio time correction value.
In an embodiment of the present application, a process for determining a sampling rate of audio frame data according to an audio-visual data stream by a timestamp correction apparatus includes: the time stamp correction device acquires a sampling rate from the audio-video data stream; or the time stamp correcting device acquires a second historical time interval between two adjacent frames of audio frame data before the current frame data in the recorded audio-video data stream; a sampling rate is determined based on the second historical time interval.
In the embodiment of the application, the audio and video data stream carries the sampling rate.
It should be noted that the number of the second historical time intervals may be one; the number of the second historical time intervals can also be multiple; i.e. the number of second historical time intervals is at least one.
In this embodiment of the application, the timestamp correction apparatus may determine the sampling rate according to at least one second historical time interval, and may obtain a first average time interval for determining an average time interval of the at least one second historical time interval; the sampling rate is then determined using equation (2):
wherein 1000 refers to time 1000 milliseconds, i.e. 1 second; 1024 is the number of points sampled for 1 second.
In the embodiment of the present application, the sampling rate may also be determined by equation (3).
Timestamp of sender = timestamp of first frame + number of frames 1000 x 1024/sample rate (3)
And S104, adjusting the time stamp of the current frame data according to the time stamp correction value to obtain the adjusted current frame data until each frame in the audio and video data stream is adjusted to have frame data with abnormal time jump.
In the embodiment of the application, after the timestamp correction device acquires the timestamp correction value, the timestamp correction device can adjust the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until each frame in the audio and video data stream is adjusted to have frame data with abnormal time jump.
In the embodiment of the present application, the current frame data includes current audio frame data and current video frame data; the process of adjusting the timestamp of the current frame data by the timestamp correction device according to the timestamp correction value to obtain the adjusted current frame data includes: the time stamp correcting device determines a system time interval between current audio frame data and current video frame data; under the condition that the system time interval is greater than the track offset threshold and less than the system time difference threshold, adjusting the timestamp correction value to obtain an adjusted timestamp correction value; and adjusting the timestamp of the current frame data according to the adjusted timestamp correction value to obtain the adjusted current frame data.
In this embodiment of the present application, the timestamp correction apparatus determines a system Time interval between current audio frame data and current video frame data, which may be a first Time for determining Universal Time Coordinated (UTC) of receiving a current audio frame; determining a second time at which the video frame receives UTC; and obtaining the system time interval by using the time difference between the first time and the second time.
In the embodiment of the present application, the track offset threshold may be a threshold configured in the timestamp correction apparatus; the track offset threshold can also be a threshold value transmitted to the timestamp correction device by other equipment; the specific way of obtaining the track offset threshold by the timestamp correction device can be determined according to actual conditions, which is not limited in the embodiment of the present application.
Illustratively, the track offset threshold may be 100 milliseconds; the track offset threshold may also be 90 milliseconds; the track offset threshold may also be other time thresholds; the specific track offset threshold value may be determined according to actual conditions, which is not limited in this embodiment of the present application.
In the embodiment of the present application, the system time difference threshold may be a threshold configured in the timestamp correction apparatus; the system time difference threshold value can also be a threshold value transmitted to the timestamp correcting device by other equipment; the specific way of obtaining the system time difference threshold by the timestamp correction device may be determined according to actual conditions, which is not limited in the embodiment of the present application.
Illustratively, the system time difference threshold may be 1 second; the system time difference threshold may also be 0.5 seconds; the system time difference threshold may be other values; the specific value of the system time difference threshold may be determined according to actual conditions, which is not limited in this application.
In this embodiment of the present application, the timestamp correction device adjusts the timestamp of the current frame data according to the adjusted timestamp correction value, so as to obtain the adjusted current frame data, and may obtain the time information of the previous frame data for the timestamp correction device, determine the sum of the time information of the previous frame data and the adjusted timestamp correction value, obtain the new time, and use the new time as the actual timestamp of the current frame data, thereby obtaining the current frame data.
In this embodiment of the present application, the timestamp correction apparatus includes a dual-rail alignment module, and when it is determined that the system time interval is greater than the track offset threshold and smaller than the system time difference threshold, the timestamp correction apparatus adjusts the timestamp correction value using the dual-rail alignment module, so as to obtain the adjusted timestamp correction value.
It should be noted that, if the system time interval is less than or equal to the track offset threshold, the timestamp of the current frame data is directly adjusted by using the timestamp correction value, so as to obtain the adjusted current frame data.
In this embodiment of the present application, the timestamp correction value includes an audio time correction value and a video time correction value, and the timestamp correction device adjusts the timestamp correction value to obtain an adjusted timestamp correction value, including: under the condition that the time of the current video frame data is less than the time of the current audio frame data, the timestamp correction device adjusts the audio time correction value by using a first preset time correction value to obtain an adjusted audio time correction value; acquiring a previous video frame correction value corresponding to previous frame video data in previous frame data; under the condition that the video time correction value is smaller than the previous video frame correction value, obtaining an adjusted video time correction value according to the previous video frame correction value and a second preset time correction value; and taking the adjusted audio time correction value and the adjusted video time correction value as adjusted timestamp correction values.
In the embodiment of the present application, the first preset time correction value may be a correction value configured in the timestamp correction apparatus; the first preset time correction value can also be a correction value transmitted to the timestamp correction device by other equipment; the specific way of obtaining the first preset time correction value by the timestamp correction device may be determined according to actual conditions, which is not limited in the embodiment of the present application.
Illustratively, the first preset time correction value may be-10 ms (minus 10 ms); the first preset time correction value can also be-20 ms; the specific first preset time correction value may be determined according to actual conditions, which is not limited in the embodiment of the present application.
In this embodiment of the application, the second preset time correction value may be a correction value configured in the timestamp correction device; the second preset time correction value can also be a correction value transmitted to the timestamp correction device by other equipment; the specific way of obtaining the second preset time correction value by the timestamp correction device may be determined according to actual conditions, which is not limited in the embodiment of the present application.
For example, the second preset time correction value may be 10ms (10 msec); the second preset time correction value can also be 20ms; the specific second preset time correction value may be determined according to actual conditions, which is not limited in this embodiment of the application.
In this embodiment, the process of obtaining the adjusted video time correction value by the timestamp correction device according to the previous video frame correction value and the second preset time correction value may be to determine the sum of the previous video frame correction value and the second preset time correction value, so as to obtain the adjusted video time correction value.
It should be noted that the adjusted time stamp correction value includes an adjusted audio time correction value and an adjusted video time correction value.
In the embodiment of the present application, in the case that the time of the current video frame data is greater than or equal to the time of the current audio frame data, the audio time correction value is not adjusted. In the case where the video temporal correction value is greater than or equal to the previous video frame correction value, no adjustment is made to the previous video frame correction value.
In this embodiment of the present application, the timestamp correction apparatus adjusts the timestamp of the current frame data according to the adjusted timestamp correction value, and obtains the adjusted current frame data, including: the timestamp correction device adjusts the current video frame data by using the adjusted video time correction value to obtain adjusted video frame data; adjusting the current audio frame data by using the adjusted audio time correction value to obtain adjusted audio frame data; and taking the adjusted video frame data and the adjusted audio frame data as the adjusted current frame data.
In the embodiment of the present application, the timestamp correction device adjusts the current video frame data by using the adjusted video time correction value to obtain the adjusted video frame data, which may be time information of the previous frame of video data; determining the sum of the time information of the previous frame of video data and the adjusted video time correction value to obtain new video time; and the new video time is used as the time stamp of the current video frame data to obtain the adjusted video frame data.
In this embodiment of the present application, the process of adjusting the current audio frame data by the timestamp correction device using the adjusted audio time correction value to obtain the adjusted audio frame data includes: acquiring time information of the previous frame of audio data; determining the sum of the time information of the previous frame of audio data and the adjusted audio time correction value to obtain new audio time; and the new audio time is taken as the time stamp of the current audio frame data to obtain the adjusted audio frame data.
And acquiring the time information of the previous frame of data, determining the sum of the time information of the previous frame of data and the adjusted time stamp correction value to obtain new time, and taking the new time as the actual time stamp of the current frame of data so as to obtain the current frame of data.
For example, if it is a video frame: under the condition that the delta (time interval) is less than 0 or the delta is more than 250 (determining that time jump abnormity exists), acquiring at least one historical time interval between two adjacent frames of data before the current frame data in the recorded audio and video data stream, determining the average delta according to the at least one historical time interval, and calculating correct _ dts = video _ frames [ preidx ]. Correct _ dts + delta; judging whether the current audio/video system time is within 1 second (the system time interval is smaller than a system time difference threshold value); if the difference value offset of the current audio and video time stamps is greater than 100 (the system time interval is greater than the track offset threshold), the audio and video needs to be aligned; it is further determined whether the offset is less than 0 (i.e. the time of the current video frame data is less than the time of the current audio frame data), and if so, correct _ dts = audio _ frames [ last ]. Correct _ dts-10 (i.e. the audio time correction value is adjusted by using the first preset time correction value, resulting in an adjusted audio time correction value); whether the video timestamp is increased progressively is further judged, and if not, the adjusted video time correction value is obtained based on the previous timestamp +10 (according to the previous video frame correction value and a second preset time correction value); if offset is greater than or equal to 0, then +10 based on the last timestamp.
In the embodiment of the present application, an exemplary timestamp correction flow diagram is shown in fig. 2:
s1, a timestamp correction device acquires current frame data and previous frame data from a received audio and video data stream, and determines an audio frame time interval according to the current audio frame data and the previous frame audio data.
It should be noted that the previous frame data includes the previous frame audio data and the previous frame video data.
And S2, the timestamp correction device determines a video frame time interval according to the current video frame data and the previous frame of video data, and takes the audio frame time interval and the video frame time interval as time intervals.
And S3, the timestamp correction device determines whether time jump abnormity exists according to the time interval.
And S4, under the condition that the time jump abnormity exists, the timestamp correction device acquires a first historical time interval between two adjacent frames of video frame data before the current frame data in the recorded audio-video data stream.
And S5, the time stamp correction device determines a video time correction value according to the first historical time interval.
S6, the timestamp correction device determines the sampling rate of the audio frame data according to the audio and video data stream; an audio temporal correction value is determined based on the sampling rate.
And S7, the time stamp correction device takes the video time correction value and the audio time correction value as time stamp correction values.
And S8, the timestamp correction device acquires a timestamp correction value.
And S9, the time stamp correcting device determines the system time interval between the current audio frame data and the current video frame data.
It should be noted that the current frame data includes current audio frame data and current video frame data.
And S10, under the condition that the system time interval is greater than the track offset threshold and smaller than the system time difference threshold, the timestamp correction device adjusts the timestamp correction value to obtain the adjusted timestamp correction value.
And S11, adjusting the timestamp of the current frame data according to the adjusted timestamp correction value to obtain the adjusted current frame data until the adjusted frame data with abnormal time jump exists in each frame in the audio and video data stream.
It can be understood that the timestamp correction device determines whether time jump abnormity exists through the time interval between the current frame data and the previous frame data acquired from the audio-video data stream, under the condition that the jump abnormity exists, the timestamp of the current frame data is adjusted according to the timestamp correction value, the adjusted current frame data is obtained, frame-level slice data with the same timestamp does not exist in the adjusted audio-video data stream, the phenomenon of slice renaming is avoided, the situations of pause and frame skipping during live stream playing are avoided, and the fluency during live data playing is improved.
Based on the same inventive concept as the above timestamp correction method, the embodiment of the present application provides a timestamp correction apparatus 1, corresponding to a timestamp correction method; fig. 3 is a schematic structural diagram of a timestamp correction apparatus according to an embodiment of the present application, where the timestamp correction apparatus 1 may include:
an obtaining unit 11, configured to obtain current frame data and previous frame data from a received audio/video data stream; acquiring a timestamp correction value under the condition that time hopping abnormity exists;
a determining unit 12, configured to determine a time interval between the current frame data and the previous frame data; determining whether time jump abnormity exists according to the time interval;
and the adjusting unit 13 is configured to adjust the timestamp of the current frame data according to the timestamp correction value to obtain adjusted current frame data until each frame of the audio-video data stream is adjusted, where the time jump is abnormal.
In some embodiments of the present application, the current frame data comprises current audio frame data and current video frame data;
the determining unit 12 is configured to determine a system time interval between the current audio frame data and the current video frame data;
the adjusting unit 13 is configured to adjust the timestamp correction value when the system time interval is greater than the track offset threshold and smaller than the system time difference threshold, so as to obtain an adjusted timestamp correction value; and adjusting the timestamp of the current frame data according to the adjusted timestamp correction value to obtain the adjusted current frame data.
In some embodiments of the present application, the timestamp correction value comprises an audio temporal correction value and a video temporal correction value;
the adjusting unit 13 is configured to adjust the audio time correction value by using a first preset time correction value when the time of the current video frame data is less than the time of the current audio frame data, so as to obtain an adjusted audio time correction value;
the acquiring unit 11 is configured to acquire a previous video frame correction value corresponding to previous video data in the previous frame data; under the condition that the video time correction value is smaller than the previous video frame correction value, obtaining an adjusted video time correction value according to the previous video frame correction value and a second preset time correction value; and taking the adjusted audio time correction value and the adjusted video time correction value as the adjusted timestamp correction value.
In some embodiments of the present application, the adjusting unit 13 is configured to adjust the current video frame data by using the adjusted video temporal correction value, so as to obtain adjusted video frame data; adjusting the current audio frame data by using the adjusted audio time correction value to obtain adjusted audio frame data; and taking the adjusted video frame data and the adjusted audio frame data as the adjusted current frame data.
In some embodiments of the present application, the obtaining unit 11 is configured to obtain a first historical time interval between two adjacent frames of video frame data before the current frame data in the recorded audio-video data stream;
the determining unit 12 is configured to determine a video temporal correction value according to the first historical time interval; determining the sampling rate of audio frame data according to the audio and video data stream; determining an audio time correction value according to the sampling rate; the video temporal correction value and the audio temporal correction value are taken as the timestamp correction value.
In some embodiments of the present application, the obtaining unit 11 is configured to obtain the sampling rate from the audio-video data stream; or acquiring a second historical time interval between two adjacent frames of audio frame data before the current frame data in the recorded audio-video data stream;
the determining unit 12 is configured to determine the sampling rate according to the second historical time interval.
In some embodiments of the present application, the determining unit 12 is configured to determine that a time jump anomaly exists if the time interval is a negative value; and determining that the time jump abnormity exists under the condition that the time interval is greater than or equal to the preset time interval.
In some embodiments of the present application, the determining unit 12 is configured to determine that there is a time jump anomaly in the audio frame data if the audio frame time interval is a negative value; and under the condition that the video frame time interval is a negative value, determining that the video frame data has time jump abnormity.
In some embodiments of the present application, the time intervals comprise audio frame time intervals and video frame time intervals;
the determining unit 12 is configured to determine that time jump abnormality exists in the audio frame data when the audio frame time interval is greater than or equal to a preset audio frame time interval; and determining that the video frame data has abnormal time jump under the condition that the video frame time interval is greater than or equal to the preset video frame time interval.
In some embodiments of the present application, the previous frame data includes a previous frame of audio data and a previous frame of video data;
the determining unit 12 is configured to determine an audio frame time interval according to the current audio frame data and the previous audio frame data; determining a video frame time interval according to the current video frame data and the previous frame video data; taking the audio frame time interval and the video frame time interval as the time interval.
In some embodiments of the present application, the apparatus further comprises a detection unit and a marking unit;
the detection unit is used for detecting whether time jump abnormity exists in adjacent data frames in the audio and video data stream;
the marking unit is used for marking the state of the first frame data with the time jump abnormity as a time jump abnormity state under the condition that the time jump abnormity exists; and taking the first frame data as the current frame data.
In practical applications, the obtaining Unit 11, the determining Unit 12, and the adjusting Unit 13 may be implemented by a processor 14 on the timestamp correction apparatus 1, specifically implemented by a CPU (Central Processing Unit), an MPU (Microprocessor Unit), a DSP (Digital Signal Processing), a Field Programmable Gate Array (FPGA), or the like; the above data storage may be realized by the memory 15 on the time stamp correction apparatus 1.
An embodiment of the present application further provides a timestamp correction apparatus 1, as shown in fig. 4, where the timestamp correction apparatus 1 includes: a processor 14, a memory 15 and a communication bus 16, the memory 15 communicating with the processor 14 through the communication bus 16, the memory 15 storing a program executable by the processor 14, the program, when executed, executing the timestamp correction method as described above through the processor 14.
In practical applications, the Memory 15 may be a volatile Memory (volatile Memory), such as a Random-Access Memory (RAM); or a non-volatile Memory (non-volatile Memory), such as a Read-Only Memory (ROM), a flash Memory (flash Memory), a Hard Disk (Hard Disk Drive, HDD) or a Solid-State Drive (SSD); or a combination of the above types of memories and provides instructions and data to the processor 14.
The present embodiment provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor 14 to implement the timestamp correction method as described above.
It can be understood that the timestamp correction device determines whether time jump abnormity exists through the time interval between the current frame data and the previous frame data acquired from the audio-video data stream, under the condition that the jump abnormity exists, the timestamp of the current frame data is adjusted according to the timestamp correction value, the adjusted current frame data is obtained, frame-level slice data with the same timestamp does not exist in the adjusted audio-video data stream, the phenomenon of slice renaming is avoided, the situations of pause and frame skipping during live stream playing are avoided, and the fluency during live data playing is improved.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.
Claims (14)
1. A method of time stamp correction, the method comprising:
acquiring current frame data and previous frame data from a received audio-video data stream, and determining a time interval between the current frame data and the previous frame data;
determining whether time jump abnormity exists according to the time interval;
acquiring a timestamp correction value under the condition that time jump abnormity exists;
and adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until the adjusted frame data with abnormal time jump exists in each frame in the audio and video data stream.
2. The method of claim 1, wherein the current frame data comprises current audio frame data and current video frame data; the adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data includes:
determining a system time interval between the current audio frame data and the current video frame data;
under the condition that the system time interval is greater than a track offset threshold and smaller than a system time difference threshold, adjusting the timestamp correction value to obtain an adjusted timestamp correction value;
and adjusting the timestamp of the current frame data according to the adjusted timestamp correction value to obtain the adjusted current frame data.
3. The method of claim 2, wherein the timestamp correction values comprise audio temporal correction values and video temporal correction values; the adjusting the timestamp correction value to obtain an adjusted timestamp correction value includes:
under the condition that the time of the current video frame data is less than the time of the current audio frame data, adjusting the audio time correction value by using a first preset time correction value to obtain an adjusted audio time correction value;
acquiring a previous video frame correction value corresponding to previous frame video data in the previous frame data;
under the condition that the video time correction value is smaller than the previous video frame correction value, obtaining an adjusted video time correction value according to the previous video frame correction value and a second preset time correction value;
and taking the adjusted audio time correction value and the adjusted video time correction value as the adjusted timestamp correction value.
4. The method of claim 2, wherein the adjusting the timestamp of the current frame data according to the adjusted timestamp correction value to obtain an adjusted current frame data comprises:
adjusting the current video frame data by using the adjusted video time correction value to obtain adjusted video frame data;
adjusting the current audio frame data by using the adjusted audio time correction value to obtain adjusted audio frame data;
and taking the adjusted video frame data and the adjusted audio frame data as the adjusted current frame data.
5. The method of claim 1, wherein prior to obtaining the timestamp correction value, the method further comprises:
acquiring a first historical time interval between two adjacent frames of video frame data before the current frame data in the recorded audio-video data stream;
determining a video temporal correction value according to the first historical time interval;
determining the sampling rate of audio frame data according to the audio and video data stream;
determining an audio time correction value according to the sampling rate;
the video temporal correction value and the audio temporal correction value are taken as the timestamp correction value.
6. The method of claim 5, wherein determining the sampling rate of the audio frame data from the stream of audiovisual data comprises:
obtaining the sampling rate from the audio-video data stream;
or acquiring a second historical time interval between two adjacent frames of audio frame data before the current frame data in the recorded audio-video data stream;
determining the sampling rate based on the second historical time interval.
7. The method of claim 1, wherein the determining whether a time hopping anomaly exists based on the time interval comprises:
determining that time jump abnormity exists under the condition that the time interval is a negative value;
and determining that the time jump abnormity exists under the condition that the time interval is greater than or equal to the preset time interval.
8. The method of claim 7, wherein the time intervals comprise an audio frame time interval and a video frame time interval; determining that a time jump anomaly exists when the time interval is a negative value, comprising:
determining that time jump abnormity exists in the audio frame data under the condition that the audio frame time interval is a negative value;
and under the condition that the video frame time interval is a negative value, determining that the video frame data has time jump abnormity.
9. The method of claim 7, wherein determining that a time jump anomaly exists if the time interval is greater than or equal to a preset time interval comprises:
determining that time jump abnormity exists in the audio frame data under the condition that the audio frame time interval is greater than or equal to a preset audio frame time interval;
and determining that the video frame data has abnormal time jump under the condition that the video frame time interval is greater than or equal to the preset video frame time interval.
10. The method of claim 1, wherein the previous frame data comprises a previous frame of audio data and a previous frame of video data; the determining the time interval between the current frame data and the previous frame data includes:
determining an audio frame time interval according to the current audio frame data and the previous audio frame data;
determining a video frame time interval according to the current video frame data and the previous frame video data;
taking the audio frame time interval and the video frame time interval as the time interval.
11. The method of claim 1, wherein before the obtaining the current frame data and the previous frame data from the received audio-visual data stream, the method further comprises:
detecting whether time jump abnormity exists in adjacent data frames in the audio and video data stream;
under the condition that time jump abnormity exists, marking the state of the first frame data with the time jump abnormity as a time jump abnormity state;
and taking the first frame data as the current frame data.
12. A time stamp correction apparatus, characterized in that the apparatus comprises:
the acquisition unit is used for acquiring current frame data and previous frame data from the received audio and video data stream; acquiring a timestamp correction value under the condition that time jump abnormity exists;
a determining unit, configured to determine a time interval between the current frame data and the previous frame data; determining whether time jump abnormity exists according to the time interval;
and the adjusting unit is used for adjusting the timestamp of the current frame data according to the timestamp correction value to obtain the adjusted current frame data until the adjusted frame data with abnormal time jump exists in each frame in the audio-video data stream.
13. A time stamp correction apparatus, characterized in that the apparatus comprises:
a memory, a processor, and a communication bus, the memory in communication with the processor through the communication bus, the memory storing a timestamp correction program executable by the processor, the timestamp correction program when executed causing the processor to perform the method of any of claims 1-11.
14. A storage medium having stored thereon a computer program for application in a timestamp correction apparatus, wherein the computer program when executed by a processor implements the method of any of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211469291.XA CN115842923A (en) | 2022-11-22 | 2022-11-22 | Timestamp correction method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211469291.XA CN115842923A (en) | 2022-11-22 | 2022-11-22 | Timestamp correction method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115842923A true CN115842923A (en) | 2023-03-24 |
Family
ID=85576319
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211469291.XA Pending CN115842923A (en) | 2022-11-22 | 2022-11-22 | Timestamp correction method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115842923A (en) |
-
2022
- 2022-11-22 CN CN202211469291.XA patent/CN115842923A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104618786B (en) | Audio and video synchronization method and device | |
| US9407387B2 (en) | Apparatus and method for synchronized transmission of multimedia content over an asynchronous network | |
| US8594184B2 (en) | Method and apparatus for controlling video-audio data playing | |
| US8402494B1 (en) | Switching content | |
| US10097874B2 (en) | System and method for monitoring media stream buffers of media output devices in order to synchronize media content output | |
| CN106470352B (en) | Live channel playing method, device and system | |
| JP5094546B2 (en) | COMMUNICATION DEVICE, COMMUNICATION METHOD, PROGRAM | |
| US8873590B2 (en) | Apparatus and method for correcting jitter | |
| CN111031385B (en) | Video playing method and device | |
| US20180367827A1 (en) | Player client terminal, system, and method for implementing live video synchronization | |
| JP6323535B2 (en) | Receiving apparatus and receiving method | |
| JP2006262288A (en) | Video data delivery server and video data delivery method | |
| CN103491430A (en) | Streaming media data processing method and electronic device | |
| CN104320676B (en) | A kind of method of program clock reference exception runout correction in video transmission stream | |
| CN115842923A (en) | Timestamp correction method and device and storage medium | |
| JP5383520B2 (en) | Relay device, program, system, and method for correcting loss of synchronization between video frame and audio frame | |
| JP2007288269A (en) | Video image receiving reproducing apparatus | |
| CN109194975B (en) | Audio and video live broadcast stream following method and device | |
| US10135945B2 (en) | Methods and systems for boundary placement in a data stream | |
| WO2005117431A1 (en) | Method for synchronising video and audio data | |
| US20150131970A1 (en) | Multimedia file playback method, playback apparatus and system | |
| CN110933516B (en) | Multimedia live broadcasting method, device and equipment | |
| JP2014135728A (en) | Video transmission system and video transmission method | |
| JP2008067179A (en) | Receiver and coded data reproduction method | |
| CA2651701A1 (en) | Generation of valid program clock reference time stamps for duplicate transport stream packets |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |