CN113382300B - Audio and video playing method and device - Google Patents
Audio and video playing method and device Download PDFInfo
- Publication number
- CN113382300B CN113382300B CN202110635135.5A CN202110635135A CN113382300B CN 113382300 B CN113382300 B CN 113382300B CN 202110635135 A CN202110635135 A CN 202110635135A CN 113382300 B CN113382300 B CN 113382300B
- Authority
- CN
- China
- Prior art keywords
- time
- clock
- current
- ctr
- stc
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4305—Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Abstract
The application discloses an audio and video playing method and device, wherein the method comprises the following steps: in the process of audio and video playing, a playing terminal carries out rendering synchronous control based on a display timestamp of a data frame to be rendered and playing elapsed time of data of a corresponding type, and carries out synchronous adjustment control on a special clock of the playing terminal based on source end time so as to match the source end time; the play elapsed time is obtained based on the dedicated clock. According to the method and the device, rendering synchronous control is performed through the special clock based on the playing terminal, synchronous adjustment control is performed on the special clock periodically based on the time of the source end, and the playing terminal can smoothly play audio and video.
Description
Technical Field
The invention relates to the technical field of multimedia, in particular to an audio and video playing method and device.
Background
With more and more TV vendors supporting the next generation of TV ATSC3.0 standard, 4K/8K high resolution high frame rate programs will become the mainstream of future TVs.
The inventor finds that when the ATSC3.0 standard is adopted, the problem that the audio and video cannot be smoothly played exists when the user terminal adopts the existing audio and video playing method. The inventors have found through earnest study analysis that the cause of the problem is as follows:
in the existing audio and video playing method, after a broadcast operator transmits audio and video signals to a user terminal, the user terminal needs to recover a clock with the frequency consistent with that of the broadcast operator, the initial value is the first program reference time received by the user terminal, and data is processed in a decoding link based on the clock. In the existing television standard (such as MPEG-2 International Standard Specification ISO/IEC 13818), the clock frequency of an audio/video source end is specifically specified to be 27MHz. Therefore, the user terminal can recover the clock which is completely consistent with the frequency of the audio and video source terminal based on the standard specification. In other systems such as the ATSC3.0 standard, the specific value of the clock frequency of the audio/video source is not determined, so that the user terminal cannot recover a clock having a frequency completely consistent with that of the audio/video source, thereby causing a difference between a reference clock of the user terminal and an encoding clock of the audio/video source. After a few hours or even longer continuous accumulation, the difference may finally cause data starvation (that is, the playing progress is faster than the data generation speed of the source end due to the reference clock being faster than the encoding clock of the audio/video source end, so that the arrival of the audio/video data needs to be waited for in the playing process) or data accumulation (that is, the playing progress is slower than the data generation speed of the source end due to the reference clock being slower than the encoding clock of the source end, so that the audio/video data is accumulated at the user terminal side), so that the audio/video cannot be played smoothly.
Disclosure of Invention
In view of the above, the main object of the present invention is to provide an audio and video playing method and apparatus, which can enable audio and video to be smoothly played.
In order to achieve the above purpose, the embodiment of the present invention provides a technical solution:
an audio and video playing method comprises the following steps:
in the audio and video playing process, the playing terminal carries out rendering synchronous control based on the display timestamp of the data frame to be rendered and the playing elapsed time of the data of the corresponding type, and carries out synchronous adjustment control on a special clock of the playing terminal based on the source end time so as to match the source end time; the play elapsed time is obtained based on the dedicated clock.
Preferably, the performing rendering synchronization control includes:
when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied pts ≤STC elapsed ≤Frame pts +D max Rendering is carried out based on the current data frame to be rendered; wherein, STC elapsed Frame for the current elapsed time of the playback pts Display time stamp for the current data frame to be rendered, D max To prepareSetting a maximum allowable delay display time; d is not less than 0 max ≤Frame duration ；Frame duration A display duration for a single data frame;
if STC is satisfied elapsed <Frame pts Then rendering is performed after waiting time Δ t based on the current data Frame to be rendered, Δ t = Frame pts -STC elapsed ；
If STC is satisfied elapsed >Frame pts +D max Then the current data frame to be rendered is discarded.
Preferably, the acquiring of the play elapsed time includes:
when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased;
calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time; the first initial time is the counting time obtained when the first data frame played by the audio and video reaches a renderer.
Preferably, the performing synchronization adjustment control on the dedicated clock comprises:
when the preset synchronous adjustment time is reached, the playing terminal acquires the current source end time SrcTime curr And the current count time STC of the special clock curr Wherein the count time STC curr And the source time SrcTime curr Are all monotonically increasing values;
calculating the count time STC curr And a second initial time STC init To obtain a first relative time STC diff (ii) a Calculating the source time SrcTime curr And a third initial time SrcTime init To obtain a second relative time SrcTime diff (ii) a Wherein the second initial time STC init The count time obtained when the first synchronization adjustment time arrives; the third initial time SrcTime init The source end time obtained when the first synchronous adjustment moment arrives is obtained;
calculating the first relative time STC diff And a second relative time SrcTime diff Obtaining the Time error Time of the special clock relative to the audio/video source end clock error ；
If the Time error Time error If the error is larger than the preset maximum error threshold, the step length is finely adjusted according to a preset first frequency, and the output frequency of the special clock is reduced; the maximum error threshold is greater than zero;
if the Time error Time error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.
Preferably, the obtaining of the current counting time of the dedicated clock comprises:
obtaining a current clock count value W from the dedicated clock ctr_curr ；
If the current last clock count value W ctr_pre If the value is the preset initial value, the last clock count value W is updated ctr_pre Counting value W for current clock ctr_curr ；
If W is satisfied ctr_curr ＜W ctr_pre Adding one to the current revolution times; the initial value of the revolution number is 0;
according to W ctr_accu ＝W ctr_curr +N×W ctr_max Calculating the cumulative count W of the clock ctr_accu (ii) a Wherein N is the current revolution number; updating the last clock count value W ctr_pre Is the current clock count value W ctr_curr ；W ctr_max A maximum clock count value for the dedicated clock;
cumulatively counting the clock by W ctr_accu And dividing the current clock frequency of the special clock to obtain the current counting time of the special clock.
The embodiment of the invention discloses an audio and video playing device, which comprises: the system comprises a special clock, a rendering control unit and a clock synchronization unit; wherein, the first and the second end of the pipe are connected with each other,
the rendering control unit is used for performing rendering synchronization control based on a display timestamp of a data frame to be rendered and the playing elapsed time of corresponding type data in the audio and video playing process, wherein the playing elapsed time is obtained based on a special clock of a playing terminal;
and the clock synchronization unit is used for carrying out synchronous adjustment control on the special clock based on the source end time in the audio and video playing process so as to match the source end time.
Preferably, the rendering control unit is specifically configured to perform the rendering synchronization control, and includes:
when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied pts ≤STC elapsed ≤Frame pts +D max Rendering is carried out based on the current data frame to be rendered; wherein, STC elapsed Frame for the current elapsed time of the playback pts Display time stamp for the current data frame to be rendered, D max Displaying the time for a preset maximum allowable delay; d is not less than 0 max ≤Frame duration ；Frame duration A display duration for a single data frame;
if STC is satisfied elapsed <Frame pts Then rendering is performed after waiting time Δ t based on the current data Frame to be rendered, Δ t = Frame pts -STC elapsed ；
If STC is satisfied elapsed >Frame pts +D max Then the current data frame to be rendered is discarded.
Preferably, the rendering control unit is specifically configured to acquire the play elapsed time, and includes:
when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased;
calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time; the first initial time is the counting time obtained when the first data frame of the audio and video playing reaches a renderer.
Preferably, the clock synchronization unit is specifically configured to perform synchronization adjustment control on the dedicated clock, and includes:
when the preset synchronous adjustment moment is reached, the current source end time SrcTime is acquired curr And the current count time STC of the special clock curr Wherein the count time STC curr And the source time SrcTime curr Are all monotonically increasing values;
calculating the count time STC curr And a second initial time STC init To obtain a first relative time STC diff (ii) a Calculating the source time SrcTime curr And a third initial time SrcTime init To obtain a second relative time SrcTime diff (ii) a Wherein the second initial time STC init The count time obtained when the first synchronization adjustment time arrives; the third initial time SrcTime init The source end time obtained when the first synchronous adjustment moment arrives is obtained;
calculating the first relative time STC diff And a second relative time SrcTime diff Obtaining the Time error Time of the special clock relative to the audio/video source end clock error ；
If the Time error Time error If the error is larger than the preset maximum error threshold, the step length is finely adjusted according to a preset first frequency, and the output frequency of the special clock is reduced; the maximum error threshold is greater than zero;
if the Time error Time error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.
Preferably, the device further comprises a dedicated timing unit;
the timing unit is used for determining the current counting time of the special clock according to the time acquisition instruction of the rendering control unit or the clock synchronization unit and feeding back the current counting time to the corresponding unit; wherein determining the current count time of the dedicated clock comprises:
obtaining a current clock count value W from the dedicated clock ctr_curr ；
If the current last clock count value W ctr_pre If the value is the preset initial value, the last clock count value W is updated ctr_pre Is the current clock count value W ctr_curr ；
If W is satisfied ctr_curr ＜W ctr_pre Adding one to the current revolution times; the initial value of the revolution number is 0;
according to W ctr_accu ＝W ctr_curr +N×W ctr_max Calculating the cumulative count W of the clock ctr_accu (ii) a Wherein N is the current revolution number; updating the last clock count value W ctr_pre Is the current clock count value W ctr_curr ；W ctr_max A maximum clock count value for the dedicated clock;
accumulating the clock by a count W ctr_accu And dividing the frequency of the special clock by the current clock frequency of the special clock to obtain the current counting time of the special clock.
The embodiment of the invention also discloses audio and video playing equipment, which comprises a processor and a memory;
the memory stores an application program executable by the processor, and the application program is used for enabling the processor to execute the audio and video playing method.
The embodiment of the invention also discloses a computer readable storage medium, in which computer readable instructions are stored, and the computer readable instructions are used for executing the audio and video playing method described above.
It can be seen from the above technical solutions that, in the audio/video playing scheme provided in the embodiments of the present invention, in the audio/video playing process, the special clock of the playing terminal is synchronously adjusted and controlled based on the source time, so that it can be ensured that the special clock of the playing terminal matches with the encoding clock of the source in the audio/video playing process, and in the rendering processing link, rendering synchronization control is performed based on the display timestamp of the data frame to be rendered and the playing elapsed time obtained based on the special clock of the playing terminal. Therefore, on one hand, smooth playing can be achieved by using the special clock matched with the time of the source end, and on the other hand, the requirements of audio and video playing with high real-time requirements such as live broadcasting and the like can be met by synchronizing in the rendering link. In addition, the special clock of the playing terminal is synchronously adjusted and controlled based on the time of the source end, so that the decoupling between the clock of the source end and the clock of the playing end can be realized, the requirement of the rendering processing link on the clock precision is lower than the requirement of the encoding clock precision of the source end, and the error which can be tolerated by the audio and video rendering is allowed to exist.
Drawings
Fig. 1 is a schematic flow diagram of an audio/video playing method according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating a method for acquiring a count time of a dedicated clock according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating a method for controlling the synchronous adjustment of a dedicated clock of a playback terminal according to an embodiment of the present invention;
fig. 4 is a structural diagram of an audio/video playing device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Considering that many programs played by broadcast operators have strong real-time performance, the conventional buffering mode is adopted for synchronous processing in the decoding stage, so that on one hand, the audio and video playing time is prolonged, the real-time performance of the audio and video playing cannot be guaranteed, and on the other hand, the smoothness of the audio and video cannot be guaranteed. Therefore, in the rendering link of audio and video playing, the rendering control is carried out based on the special clock of the user terminal, and the special clock is periodically and synchronously adjusted based on the time of the source end in the playing process, so that the speed of playing the audio and video by the receiving end is consistent with the speed of generating the audio and video by the source end, and the effect of smoothly playing the audio and video is achieved.
Fig. 1 is a schematic flow diagram of an audio and video playing method according to an embodiment of the present invention, and as shown in fig. 1, the embodiment mainly includes:
In this step, after the playing terminal starts the audio/video playing process, rendering synchronization control is performed on each data frame that enters the rendering link after being processed by the demultiplexing and audio/video decoder, so as to achieve the effect of smooth playing.
Compared with the traditional cache synchronization method adopted in the decoding link, the method adopts the synchronization control of the rendering link, so that the real-time performance of audio and video playing can be met, and the smooth effect of the audio and video playing can be realized.
The play elapsed time is used to characterize: in the process of audio and video playing, after a first audio frame or video frequency reaches a renderer of a playing terminal for rendering processing, the playing duration of corresponding type data is recorded based on the time of a special clock.
Preferably, in one embodiment, the playing elapsed time may be obtained by:
when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased; and calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time.
The first initial time is the counting time obtained when the first data frame with the same type as the data frame to be rendered reaches a renderer in the audio and video playing process. Thus, the initial value of the play elapsed time is zero. Since the count time of the dedicated clock is an increment time value, the play elapsed time is a time that is incremented from zero.
Preferably, in one embodiment, in order to accurately obtain the counting time of the dedicated clock, as shown in fig. 2, the following method may be adopted to obtain the counting time:
step a1, obtaining the current clock count value W from the special clock of the playing terminal ctr_curr 。
It should be noted that, in this step, the dedicated clock generates pulses and counts by hardware, and the count value is divided by the frequency value to obtain the corresponding count time. Clock hardware typically uses a limited number of register bits to store the clock count. Assuming that the number of register bits is M and the Clock frequency is Clock Freq Then the maximum time that can be represented is (2) M -1)/Clock Freq The maximum value is reached and the slew starts again from 0, e.g. 33 bits in register, 90KHz for example, and the maximum value is (2) 33 -1)/90000, about 95,443 seconds, 26.5 hours. In order to make the count time of the dedicated clock monotonically increase, the above-described rotation needs to be handled when performing the count time, that is, the number of rotations needs to be considered when performing the count time.
Step a2, if the current previous clock count value W ctr_pre If the value is the preset initial value, the last clock count value W is updated ctr_pre Is the current clock count value W ctr_curr 。
This step is to count the value W of the previous clock when the count time is first obtained ctr_pre Updated to the current clock count value W ctr_curr 。
Specifically, the W ctr_pre The initial value of (b) can be set by a person skilled in the art according to actual needs, for example, but not limited thereto, can be zero.
Step a3, if W is satisfied ctr_curr ＜W ctr_pre Then the current number of revolutions is incremented by one.
The initial value of the number of revolutions is 0.
Here, if W ctr_curr ＜W ctr_pre If so, it indicates that the counting revolution is currently generated, so one needs to be added to the number of revolutions.
Step a4, according toW ctr_accu ＝W ctr_curr +N×W ctr_max Calculating the cumulative count W of the clock ctr_accu 。
Wherein N is the current revolution number; updating the last clock count value W ctr_pre Is the current clock count value W ctr_curr ；W ctr_max Is the maximum clock count value of the dedicated clock.
Here, count W is accumulated in the calculation clock ctr_accu In time, the current revolution number is considered, so that the counting time of the special clock is ensured to be monotonically increased.
Step a5, accumulating the clock into a count W ctr_accu And dividing the current clock frequency of the special clock to obtain the current counting time of the special clock.
The Presentation Time Stamp (PTS) of the data Frame may specifically adopt an existing calculation method, that is, according to the Frame pts ＝(n-1)×Frame duration And calculating to obtain the display time stamp Frame of the data Frame pts And n is the number of the audio data frame or the video data frame, and the number is the number of the data frame of the same type, namely the nth audio data frame or the nth video data frame is represented. Taking video Frame data as an example, the display timestamp of the first video Frame is 0, and the subsequent video frames pass through the Frame duration And accumulating. For example, for 60fps video, frame duration 16.667ms, frame of the first video Frame pts 0ms, second video Frame pts 16.667ms, the Nth Frame _ pts is (N-1). Times. 16.667ms, and therefore, the Frame pts Is a time value incremented from 0. Thus, the display time stamp Frame can be displayed pts And comparing the elapsed time of the playing, which is also increased from zero, to know whether the time for the data frame to be rendered to reach the renderer is matched with the playing speed.
Specifically, in step 101, the display time stamp of the data frame to be rendered is compared with the play elapsed time of the corresponding type of data, and rendering synchronization control is performed according to the comparison result.
In order to obtain a better smooth playing effect, in one embodiment, the following method may be adopted for rendering synchronization control:
when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied pts ≤STC elapsed ≤Frame pts +D max Rendering is carried out based on the current data frame to be rendered; wherein, STC elapsed Frame for the current elapsed time of the playback pts Display time stamp for the current data frame to be rendered, D max Displaying the time for a preset maximum allowable delay; d is more than or equal to 0 max ≤Frame duration ；Frame duration A display duration for a single data frame;
if STC is satisfied elapsed <Frame pts Then rendering is performed after waiting time Δ t based on the current data Frame to be rendered, Δ t = Frame pts -STC elapsed ；
If STC is satisfied elapsed >Frame pts +D max And discarding the current data frame to be rendered.
The method can be used for respectively realizing the smooth playing of the audio frame data and the smooth playing of the video frame data. Wherein, if the Frame is satisfied pts ≤STC elapsed ≤Frame pts +D max If the current time meets the requirement of smooth playing, the data frame to be rendered is suitable for rendering, and therefore, the data frame to be rendered is immediately rendered based on the current data frame to be rendered.
Said D max Satisfies the following conditions: d is not less than 0 max ≤Frame duration D can be set by a person skilled in the art in particular according to the actual smoothing requirements max Preferably, in order to reduce the discarding of the data frame while satisfying the smooth display of the picture, D may be set max ＝Frame duration 。
If STC is satisfied elapsed <Frame pts Then, it means that the current data Frame to be rendered arrives at the renderer earlier than the smooth playing speed, so it is necessary to wait for a certain interval (i.e. Frame) pts -STC elapsed ) Then, rendering can be performed based on the current data frame to be rendered to achieve smooth playingAnd (5) effect.
If STC is satisfied elapsed >Frame pts +D max If the data frame to be rendered reaches the renderer at a speed later than the smooth playing speed, the data frame to be rendered currently misses the playing time corresponding to the smooth playing requirement, and therefore the data frame to be rendered currently needs to be discarded.
In the above method, frame is used pts ≤STC elapsed ≤Frame pts +D max The method is suitable for the current instant rendering opportunity, so that certain tolerable error is allowed between the special clock of the playing terminal and the source end clock, the precision of the special clock is not required to be completely consistent with that of the source end coding clock, the precision reaches the precision of millisecond or above, namely, the clock frequency is more than or equal to 1KHz, smooth audio and video playing can be realized, the precision requirement of the special clock can be reduced, the power consumption and the production cost of the special clock are further reduced, and the applicability of the method is improved.
And 102, in the audio and video playing process, the playing terminal synchronously adjusts and controls the special clock based on the source time so as to match the source time.
In this step, in order to ensure the accuracy of rendering synchronization control performed based on the dedicated clock of the play terminal in step 101, in the audio/video playing process, the dedicated clock is synchronously adjusted and controlled based on the source end time to match the source end time, that is, the dedicated clock is matched with the encoding clock of the audio/video source end.
The special clock of the playing terminal is a reference clock which is synchronized in the rendering link, and each data frame has a certain display duration, so that the playing time of the data frame has a certain tolerable floating range, and thus, the precision requirement of the special clock of the playing terminal does not need to have a high precision requirement like a coding clock of a source end, and only the time error between the special clock and the clock of the source end does not influence the watching experience of human eyes.
Preferably, in order to make the dedicated clock better synchronized with the source clock, in one embodiment, as shown in fig. 3, the following method can be adopted to timely perform synchronization adjustment control on the dedicated clock of the cast terminal:
Wherein the count time STC curr And the source time SrcTime curr Are all monotonically increasing values, so that the relative times respectively calculated by the subsequent steps based on the two parameters are also both increasing values.
In this step, when each synchronization adjustment time arrives, the current source end time SrcTime needs to be acquired simultaneously curr And the current counting time of the special clock is used for further determining the time error of the current special clock relative to the audio and video source end clock in the subsequent steps based on the two currently obtained times, and further determining whether the special clock needs to be synchronously adjusted according to the time error so as to enable the time of the special clock to be matched with the source end time.
In practical applications, a person skilled in the art may set the synchronization adjustment time according to an actual timing adjustment policy, for example, the synchronization adjustment time may be determined according to a preset synchronization adjustment period. For the synchronization adjustment period, if the setting is too long, timeliness of the synchronization adjustment cannot be satisfied, and if the setting is too short, too much control overhead is generated, specifically, a suitable period length, such as 10s, may be set by a person skilled in the art in consideration of an actual error time accumulation rate of the dedicated clock and the source-end clock, according to a policy of reducing the control overhead as much as possible while the timeliness of the synchronization adjustment is satisfied, but not limited thereto.
Here, the current counting time of the dedicated clock may be obtained by the same method as that in step 101, and is not described herein again.
It should be noted that the source time is sent to the play terminal by the source at a fixed time, so that the play terminal can obtain the source time from the source signal by analyzing.
Wherein the second initial time STC init The count time obtained when the first synchronization adjustment time arrives; the third initial time SrcTime init And the source end time obtained when the first synchronous adjustment moment arrives is adjusted.
In this step, the first relative time STC diff Indicating the time interval between the counting time of the currently acquired dedicated clock and the counting time acquired when the first synchronization adjustment time arrives, the second relative time SrcTime diff And the time interval between the currently acquired source time and the acquired source time when the first synchronous adjustment time arrives is represented. The difference between the first relative time and the second relative time may reflect whether the private clock matches the source clock.
Here, if the dedicated clock matches the source clock, the difference between the first relative Time and the second relative Time (i.e., time error Time) error ) Will be small, otherwise, the difference between the two will be large.
if the Time error Time error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.
Here, if the Time error Time error If the maximum error threshold is greater than the preset maximum error threshold and the maximum error threshold is greater than zero, it indicates that the dedicated clock is faster relative to the source-end clock, and at this time, the output frequency of the dedicated clock needs to be reduced, so that the timing of the dedicated clock is slowed down, and the dedicated clock is matched with the source-end clock. If Time error Time error If the minimum error threshold is smaller than the preset minimum error threshold and is smaller than zero, the special clock is slow relative to the source end clock, and at the moment, the output frequency of the special clock needs to be increased, so that the special clock is accelerated to be matched with the source end clock. If Time error Time error When the minimum error threshold value and the maximum error threshold value are within the range, the current special clock is matched with the source end clock, and the frequency of the special clock does not need to be adjusted.
Specifically, the first frequency fine tuning step is used to control the amplitude of the output frequency of the dedicated clock that is decreased each time, and the second frequency fine tuning step is used to control the amplitude of the output frequency of the dedicated clock that is increased each time.
For the minimum error threshold and the maximum error threshold, those skilled in the art can set the matching degree requirement of the dedicated clock and the source clock in practical application. Preferably, the absolute values of the minimum error threshold and the maximum error threshold are the same, but not limited thereto.
The method embodiment can show that the audio and video playing method can synchronously adjust and control the special clock of the playing terminal based on the time of the source terminal in the audio and video playing process, can ensure that the special clock of the playing terminal is matched with the coding clock of the source terminal, and can perform rendering synchronization control based on the display time stamp of the data frame to be rendered and the playing elapsed time of the corresponding type of data obtained based on the special clock of the playing terminal in the rendering processing link. Therefore, on one hand, smooth playing can be achieved by using the special clock matched with the time of the source end, and on the other hand, the requirements of audio and video playing with high real-time requirements such as live broadcasting and the like can be met by synchronizing in the rendering link. In addition, the special clock of the playing terminal is synchronously adjusted and controlled based on the time of the source end, so that the decoupling between the clock of the source end and the clock of the playing end can be realized, the requirement of the rendering processing link on the clock precision is lower than the requirement of the encoding clock precision of the source end, and the error which can be tolerated by the audio and video rendering is allowed to exist.
Corresponding to the above embodiment of the audio and video playing method, the embodiment of the present invention further discloses an audio and video playing device, as shown in fig. 4, the device at least includes: a rendering control unit 401, a dedicated clock 402, and a clock synchronization unit 403; wherein, the first and the second end of the pipe are connected with each other,
the rendering control unit 401 is configured to perform rendering synchronization control based on a display timestamp of a data frame to be rendered and a playing elapsed time of data of a corresponding type in an audio/video playing process, where the playing elapsed time is obtained based on a dedicated clock of a playing terminal.
The clock synchronization unit 403 is configured to perform synchronization adjustment control on the dedicated clock based on the source time in the audio/video playing process, so as to match the source time.
In an embodiment, the rendering control unit 401 is specifically configured to perform the rendering synchronization control, and includes:
when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied pts ≤STC elapsed ≤Frame pts +D max Rendering is carried out based on the current data frame to be rendered; wherein, STC elapsed Frame for the current elapsed time of the playback pts Display time stamp for the current data frame to be rendered, D max Displaying the time for a preset maximum allowable delay; d is not less than 0 max ≤Frame duration ；Frame duration A display duration for a single data frame;
if STC is satisfied elapsed <Frame pts Rendering based on the current data Frame to be rendered after waiting time Δ t, Δ t = Frame pts -STC elapsed ；
If STC is satisfied elapsed >Frame pts +D max And discarding the current data frame to be rendered.
In an embodiment, the rendering control unit 401, specifically configured to obtain the playing elapsed time, includes:
when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased;
calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time; the first initial time is the counting time obtained when the first data frame of the audio and video playing reaches a renderer.
In an embodiment, the clock synchronization unit 402 is specifically configured to perform synchronization adjustment control on the dedicated clock, and includes:
when the preset synchronous adjustment moment is reached, the current source end time SrcTime is acquired curr And the current count time STC of the special clock curr Wherein the count time STC curr And the source time SrcTime curr Are all monotonically increasing values;
calculating the count time STC curr And a second initial time STC init To obtain a first relative time STC diff (ii) a Calculating the source time SrcTime curr And a third initial time SrcTime init To obtain a second relative time SrcTime diff (ii) a Wherein the second initial time STC init Obtained when the first synchronization adjustment moment arrivesThe counting time; the third initial time SrcTime init The source end time obtained when the first synchronous adjustment moment arrives is obtained;
calculating the first relative time STC diff And a second relative time SrcTime diff Obtaining the Time error Time of the special clock relative to the audio/video source end clock error ；
If the Time error Time error If the error is larger than the preset maximum error threshold, the step length is finely adjusted according to a preset first frequency, and the output frequency of the special clock is reduced; the maximum error threshold is greater than zero;
if the Time error Time error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.
In one embodiment, the apparatus further comprises a dedicated timing unit 404;
the timing unit 404 is configured to determine a current counting time of the dedicated clock according to a time obtaining instruction of the rendering control unit or the clock synchronization unit, and feed back the current counting time to a corresponding unit; wherein determining the current count time of the dedicated clock comprises:
obtaining a current clock count value W from the dedicated clock ctr_curr ；
If the current last clock count value W ctr_pre If the value is the preset initial value, the last clock count value W is updated ctr_pre Is the current clock count value W ctr_curr ；
If W is satisfied ctr_curr ＜W ctr_pre Adding one to the current revolution times; the initial value of the revolution number is 0;
according to W ctr_accu ＝W ctr_curr +N×W ctr_max Calculating the cumulative count W of the clock ctr_accu (ii) a Wherein N is the current revolution number; updating the last clock count value W ctr_pre Is the current clock count value W ctr_curr ；W ctr_max To the special purposeCounting a value with a maximum clock of a clock;
accumulating the clock by a count W ctr_accu And dividing the frequency of the special clock by the current clock frequency of the special clock to obtain the current counting time of the special clock.
Corresponding to the embodiment of the audio and video playing method, the embodiment of the invention also discloses audio and video playing equipment, which comprises a processor and a memory;
the memory stores an application program executable by the processor, and the application program is used for enabling the processor to execute the audio and video playing method.
The memory may be embodied as various storage media such as an Electrically Erasable Programmable Read Only Memory (EEPROM), a Flash memory (Flash memory), and a Programmable Read Only Memory (PROM). The processor may be implemented to include one or more central processors or one or more field programmable gate arrays, wherein the field programmable gate arrays integrate one or more central processor cores. In particular, the central processor or central processor core may be implemented as a CPU or MCU.
It should be noted that not all steps and modules in the above flows and structures are necessary, and some steps or modules may be omitted according to actual needs. The execution order of the steps is not fixed and can be adjusted as required. The division of each module is only for convenience of describing adopted functional division, and in actual implementation, one module may be divided into multiple modules, and the functions of multiple modules may also be implemented by the same module, and these modules may be located in the same device or in different devices.
The hardware modules in the various embodiments may be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (e.g., a special purpose processor such as an FPGA or ASIC) for performing specific operations. A hardware module may also comprise programmable logic devices or circuits (e.g., including a general-purpose processor or other programmable processor) that are temporarily configured by software to perform certain operations. The implementation of the hardware module in a mechanical manner, or in a dedicated permanent circuit, or in a temporarily configured circuit (e.g., configured by software), may be determined based on cost and time considerations.
The embodiment of the invention also discloses a computer readable storage medium, wherein computer readable instructions are stored, and the computer readable instructions are used for executing the audio and video playing method.
Specifically, a system or an apparatus equipped with a storage medium on which a software program code that realizes the functions of any of the embodiments described above is stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program code stored in the storage medium. Further, part or all of the actual operations may be performed by an operating system or the like operating on the computer by instructions based on the program code. The functions of any of the above-described embodiments may also be implemented by writing the program code read out from the storage medium to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then causing a CPU or the like mounted on the expansion board or the expansion unit to perform part or all of the actual operations based on the instructions of the program code.
Embodiments of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer or the cloud by a communication network.
"exemplary" means "serving as an example, instance, or illustration" herein, and any illustration, embodiment, or steps described as "exemplary" herein should not be construed as a preferred or advantageous alternative. For the sake of simplicity, the drawings are only schematic representations of the parts relevant to the invention, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and understandable, components having the same structure or function in some of the drawings are only schematically illustrated or only labeled. In this document, "a" does not mean that the number of the relevant portions of the present invention is limited to "only one", and "a" does not mean that the number of the relevant portions of the present invention "more than one" is excluded. In this document, "upper", "lower", "front", "rear", "left", "right", "inner", "outer", and the like are used only to indicate relative positional relationships between relevant portions, and do not limit absolute positions of the relevant portions.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. An audio/video playing method is characterized by comprising the following steps:
in the audio and video playing process, the playing terminal carries out rendering synchronous control based on the display timestamp of the data frame to be rendered and the playing elapsed time of the data of the corresponding type, and carries out synchronous adjustment control on a special clock of the playing terminal based on the source end time so as to match the source end time; the play elapsed time is obtained based on the dedicated clock; the special clock generates pulses through hardware and counts the pulses, and the count value is divided by the frequency value to obtain corresponding counting time;
wherein the acquiring of the play elapsed time includes:
when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased;
calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time; the first initial time is the counting time obtained when the first data frame of the audio and video playing reaches a renderer;
the step of performing synchronous adjustment control on the dedicated clock of the play terminal includes:
when the preset synchronous adjustment time arrives, the broadcastThe playing terminal obtains the current source terminal time SrcTime curr And the current count time STC of the special clock curr Wherein the count time STC curr And the source time SrcTime curr Are all monotonically increasing values;
calculating the count time STC curr And a second initial time STC init To obtain a first relative time STC diff (ii) a Calculating the source time SrcTime curr And a third initial time SrcTime init To obtain a second relative time SrcTime diff (ii) a Wherein the second initial time STC init The count time obtained when the first synchronization adjustment time arrives; the third initial time SrcTime init The source end time obtained when the first synchronous adjustment moment arrives is obtained;
calculating the first relative time STC diff And a second relative time SrcTime diff Obtaining the Time error Time of the special clock relative to the audio/video source end clock error ；
If the Time error Time error If the error is larger than the preset maximum error threshold, the step length is finely adjusted according to a preset first frequency, and the output frequency of the special clock is reduced; the maximum error threshold is greater than zero;
if the Time error Time error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.
2. The method according to claim 1, wherein the performing rendering synchronization control includes:
when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied pts ≤STC elapsed ≤Frame pts +D max Rendering is carried out based on the current data frame to be rendered; wherein, STC elapsed Frame for the current elapsed time of the playback pts Display time stamp for the current data frame to be rendered, D max Displaying the time for a preset maximum allowable delay; d is not less than 0 max ≤Frame duration ；Frame duration A display duration for a single data frame;
if STC is satisfied elapsed <Frame pts Then rendering is performed after waiting time Δ t based on the current data Frame to be rendered, Δ t = Frame pts -STC elapsed ；
If STC is satisfied elapsed >Frame pts +D max Then the current data frame to be rendered is discarded.
3. The method of claim 1, wherein obtaining the current count time of the dedicated clock comprises:
obtaining a current clock count value W from the dedicated clock ctr_curr ；
If the current last clock count value W ctr_pre If the value is the preset initial value, the last clock count value W is updated ctr_pre Is the current clock count value W ctr_curr ；
If W is satisfied ctr_curr ＜W ctr_pre Adding one to the current revolution times; the initial value of the revolution number is 0;
according to W ctr_accu ＝W ctr_curr +N×W ctr_max Calculating the cumulative count W of the clock ctr_accu (ii) a Wherein N is the current revolution number; updating the last clock count value W ctr_pre Is the current clock count value W ctr_curr ；W ctr_max A maximum clock count value for the dedicated clock;
cumulatively counting the clock by W ctr_accu And dividing the current clock frequency of the special clock to obtain the current counting time of the special clock.
4. An audio-video playback device, comprising: the system comprises a special clock, a rendering control unit and a clock synchronization unit; wherein the content of the first and second substances,
the rendering control unit is used for performing rendering synchronization control based on a display timestamp of a data frame to be rendered and the playing elapsed time of corresponding type data in the audio and video playing process, wherein the playing elapsed time is obtained based on a special clock of a playing terminal; the special clock generates pulses through hardware and counts the pulses, and the count value is divided by the frequency value to obtain corresponding counting time;
the clock synchronization unit is used for carrying out synchronous adjustment control on the special clock based on source end time in the audio and video playing process so as to match the source end time;
the rendering control unit is specifically configured to acquire the play elapsed time, and includes:
when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased;
calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time; the first initial time is the counting time obtained when the first data frame of the audio and video playing reaches a renderer;
the clock synchronization unit is specifically configured to perform synchronization adjustment control on a dedicated clock of the playback terminal, and includes:
when the preset synchronous adjustment moment is reached, the current source end time SrcTime is acquired curr And the current count time STC of the special clock curr Wherein the count time STC curr And the source time SrcTime curr Are all monotonically increasing values;
calculating the count time STC curr And a second initial time STC init To obtain a first relative time STC diff (ii) a Calculating the source time SrcTime curr And a third initial time SrcTime init To obtain a second relative time SrcTime diff (ii) a Wherein the second initial time STC init The count time obtained when the first synchronization adjustment time arrives; the third initial time SrcTime init The source end time obtained when the first synchronous adjustment moment arrives is obtained;
calculating the first relative time STC diff And a second relative time SrcTime diff Obtaining the Time error Time of the special clock relative to the audio/video source end clock error ；
If the Time error Time error If the error is larger than the preset maximum error threshold, the step length is finely adjusted according to a preset first frequency, and the output frequency of the special clock is reduced; the maximum error threshold is greater than zero;
if the Time error Time error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.
5. The apparatus according to claim 4, wherein the rendering control unit, in particular configured to perform the rendering synchronization control, comprises:
when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied pts ≤STC elapsed ≤Frame pts +D max Rendering is carried out based on the current data frame to be rendered; wherein, STC elapsed Frame for the current elapsed time of the playback pts Display time stamp for the current data frame to be rendered, D max Displaying the time for a preset maximum allowable delay; d is not less than 0 max ≤Frame duration ；Frame duration A display duration for a single data frame;
if STC is satisfied elapsed <Frame pts Rendering based on the current data Frame to be rendered after waiting time Δ t, Δ t = Frame pts -STC elapsed ；
If STC is satisfied elapsed >Frame pts +D max And discarding the current data frame to be rendered.
6. The apparatus of claim 4, further comprising a dedicated timing unit;
the timing unit is used for determining the current counting time of the special clock according to the time acquisition instruction of the rendering control unit or the clock synchronization unit and feeding back the current counting time to the corresponding unit; wherein determining the current count time of the dedicated clock comprises:
obtaining a current clock count value W from the dedicated clock ctr_curr ；
If the current last clock count value W ctr_pre If the value is the preset initial value, the last clock count value W is updated ctr_pre Counting value W for current clock ctr_curr ；
If W is satisfied ctr_curr ＜W ctr_pre Adding one to the current revolution times; the initial value of the revolution number is 0;
according to W ctr_accu ＝W ctr_curr +N×W ctr_max Calculating the cumulative count W of the clock ctr_accu (ii) a Wherein N is the current revolution number; updating the last clock count value W ctr_pre Counting value W for current clock ctr_curr ；W ctr_max A maximum clock count value for the dedicated clock;
accumulating the clock by a count W ctr_accu And dividing the current clock frequency of the special clock to obtain the current counting time of the special clock.
7. An audio and video playing device is characterized by comprising a processor and a memory;
the memory stores an application program executable by the processor, and the application program is used for causing the processor to execute the audio and video playing method according to any one of claims 1 to 3.
8. A computer-readable storage medium having stored therein computer-readable instructions for executing the audio-video playback method according to any one of claims 1 to 3.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110635135.5A CN113382300B (en) | 2021-06-08 | 2021-06-08 | Audio and video playing method and device |
| PCT/KR2022/008070 WO2022260423A1 (en) | 2021-06-08 | 2022-06-08 | Audio or video playing method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110635135.5A CN113382300B (en) | 2021-06-08 | 2021-06-08 | Audio and video playing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113382300A CN113382300A (en) | 2021-09-10 |
| CN113382300B true CN113382300B (en) | 2023-03-21 |
Family
ID=77576258
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110635135.5A Active CN113382300B (en) | 2021-06-08 | 2021-06-08 | Audio and video playing method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113382300B (en) |
| WO (1) | WO2022260423A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113382300B (en) * | 2021-06-08 | 2023-03-21 | 三星电子（中国）研发中心 | Audio and video playing method and device |
| CN114189737A (en) * | 2021-12-06 | 2022-03-15 | 国微集团(深圳)有限公司 | Digital television and rapid channel switching method thereof |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8234395B2 (en) * | 2003-07-28 | 2012-07-31 | Sonos, Inc. | System and method for synchronizing operations among a plurality of independently clocked digital data processing devices |
| KR101841313B1 (en) * | 2010-09-22 | 2018-03-22 | 톰슨 라이센싱 | Methods for processing multimedia flows and corresponding devices |
| CN103747332B (en) * | 2013-12-25 | 2018-08-10 | 乐视致新电子科技（天津）有限公司 | A kind of smoothing processing method and device of video |
| GB2536318B (en) * | 2015-09-09 | 2018-02-14 | Imagination Tech Ltd | Synchronising devices |
| US10098082B2 (en) * | 2015-12-16 | 2018-10-09 | Sonos, Inc. | Synchronization of content between networked devices |
| CN105898504A (en) * | 2016-04-26 | 2016-08-24 | 乐视控股（北京）有限公司 | Audio and video synchronization method and apparatus |
| CN108243350B (en) * | 2016-12-26 | 2020-05-05 | 深圳市中兴微电子技术有限公司 | Audio and video synchronization processing method and device |
| CN107509100A (en) * | 2017-09-15 | 2017-12-22 | 深圳国微技术有限公司 | Audio and video synchronization method, system, computer installation and computer-readable recording medium |
| CN113382300B (en) * | 2021-06-08 | 2023-03-21 | 三星电子（中国）研发中心 | Audio and video playing method and device |
-
2021
- 2021-06-08 CN CN202110635135.5A patent/CN113382300B/en active Active
-
2022
- 2022-06-08 WO PCT/KR2022/008070 patent/WO2022260423A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| CN113382300A (en) | 2021-09-10 |
| WO2022260423A1 (en) | 2022-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109714634B (en) | Decoding synchronization method, device and equipment for live data stream | |
| US6429902B1 (en) | Method and apparatus for audio and video end-to-end synchronization | |
| US7471337B2 (en) | Method of audio-video synchronization | |
| CN113382300B (en) | Audio and video playing method and device | |
| EP3334175A1 (en) | Streaming media and caption instant synchronization displaying and matching processing method, device and system | |
| CN109729373B (en) | Streaming media data mixing method and device, storage medium and computer equipment | |
| US8931024B2 (en) | Receiving apparatus and subtitle processing method | |
| US8245262B2 (en) | System and method for synchronization of television signals associated with multiple broadcast networks | |
| US10341672B2 (en) | Method and system for media synchronization | |
| EP2227910B1 (en) | Reducing media stream delay through independent decoder clocks | |
| CN106612452B (en) | method and device for synchronizing audio and video of set top box | |
| US11812103B2 (en) | Dynamic playout of transition frames while transitioning between playout of media streams | |
| JP6313704B2 (en) | Reception device and synchronization processing method thereof | |
| US20200059692A1 (en) | Dynamic Reduction in Playout of Replacement Content to Help Align End of Replacement Content with End of Replaced Content | |
| EP2866458B1 (en) | Reception device, and synchronous processing method therefor | |
| US9854295B2 (en) | Content output device and program | |
| TWI767228B (en) | Method of volume adjustment carried out in connection with content modification, content-presentation device, and non-transitory computer-readable storage medium | |
| CN102413335A (en) | Manual adjustment device and method for program audio and video synchronization | |
| US20220046335A1 (en) | Live Broadcast IP Latency Compensation | |
| KR20230086585A (en) | Method and apparatus for minimizing initial screen output delay in channel selection of digital broadcasting receiver | |
| JP2000092488A (en) | Video output phase control for decoder |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |