WO2022260423A1

WO2022260423A1 - Audio or video playing method and apparatus

Info

Publication number: WO2022260423A1
Application number: PCT/KR2022/008070
Authority: WO
Inventors: Xianghu CHEN; Yong Zhang; Ye Sun
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2021-06-08
Filing date: 2022-06-08
Publication date: 2022-12-15
Also published as: CN113382300B; CN113382300A

Abstract

The present disclosure provides an audio or video playing method and apparatus, in which the method includes: in a process of audio or video playback, performing, by a playback terminal, rendering synchronization control based on a representation timestamp of a to-be-rendered data frame and an elapsed playback time of a type of data corresponding to the to-be-rendered data frame, and performing synchronization adjustment control on a dedicated clock of the playback terminal based on a source time, so as to match the dedicated clock with the source time; in which the elapsed playback time is obtained based on the dedicated clock. In the present disclosure, the rendering synchronization control is performed based on the dedicated clock of the playback terminal, and the synchronization adjustment control is periodically performed on the dedicated clock based on source times, which enables the playback terminal to play audio or video smoothly.

Description

AUDIO OR VIDEO PLAYING METHOD AND APPARATUS

The present disclosure relates to multimedia, and in particular relates to an audio or video playing method and apparatus.

With more and more TV manufacturers supporting the next generation of TV standard, ATSC3.0 (Advanced Television Systems Committee 3.0), TV programs having a high frame rate of 4K/8K high resolution are becoming the mainstream of future TVs.

In the process of developing the present invention, the inventor finds that if adopting the ATSC3.0 standard, when a UE uses an existing audio or video playing method, audio or video cannot be played smoothly. After careful analysis, the inventor finds that the reasons for this issue are as follows:

In the existing audio or video playing method, after a broadcast operator transmits audio or video signals to user equipment (UE), the UE is required to recover a clock of the same frequency as that of the broadcast operator, an initial value of the clock being the first program reference time received by the UE, and the UE processes data in a decoding stage based on the clock. In an existing TV standard (e.g., MPEG-2 International Standard ISO/IEC 13818), it is clearly specified that an audio or video source clock frequency is 27MHz. Thus, UE may recover a clock having a frequency totally identical to that at the audio or video source. However, in other systems such as ATSC3.0, the value of the audio or video source clock frequency is not specified. In this case, UE will not be able to recover a clock having the same frequency as that at the audio or video source, which results in that there is a difference between a reference time at the UE and a coding clock at the audio or video source. This difference, after a few hours or an even longer period of continuous accumulation, will eventually lead to starvation of data (that is, since the reference clock is faster than the coding clock at the audio or video source, it leads to that the playing progress is faster than generating data at the audio or video source, which leads to that the UE needs to wait for the arrival of audio or video data during the playback process) or backlog of data (that is, since the reference clock is slower than the coding clock at the audio or video source, it leads to that the playing progress is slower than generating data at the audio or video source, which leads to backlog of audio or video data at the UE) at the UE, and causes the audio or the video unable to be smoothly played.

In view of this, an object of the present disclosure is to provide an audio or video playing method and apparatus, to enable audio or video to be smoothly played.

To achieve the above object, the technical solutions provided by the embodiments of the present disclosure are as follows.

An audio or video playing method includes:

in a process of audio or video playback, performing, by a playback terminal, rendering synchronization control based on a representation timestamp of a to-be-rendered data frame and an elapsed playback time of a type of data corresponding to the to-be-rendered data frame, and performing synchronization adjustment control on a dedicated clock of the playback terminal based on a source time, so as to match the dedicated clock with the source time; in which the elapsed playback time is obtained based on the dedicated clock.

Preferably, performing the rendering synchronization control includes:

when the to-be-rendered data frame arrives at a renderer, if Frame_pts≤STC_elapsed≤Frame_pts+D_max is satisfied, then performing rendering based on the current to-be-rendered data frame; in which STC_elapsedis currently the elapsed playback time, Frame_pts is the representation timestamp of the current to-be-rendered data frame, and D_max is a preset maximum allowable delayed representation time; 0≤D_max≤Frame_duration; in which Frame_duration is a representation duration of a single data frame;

if STC_elapsed<Frame_pts is satisfied, then after waiting for a period of time △t, performing the rendering based on the current to-be-rendered data frame, in which △t=Frame_pts-STC_elapsed; and

if STC_elapsed>Frame_pts+ D_max is satisfied, then discarding the current to-be-rendered data frame.

Preferably obtaining the elapsed playback time includes:

when the to-be-rendered data frame arrives at a renderer, obtaining a current counted time of the dedicated clock, in which counted times of the dedicated clock are monotone increasing values; and

calculating a difference between the counted time and a first initial time to obtain the elapsed playback time; in which the first initial time is a counted time obtained when a first data frame of the audio or video playback arrived at the renderer.

Preferably, performing the synchronization adjustment control on the dedicated clock of the playback terminal includes:

when a preset synchronization adjustment time is reached, obtaining, by the playback terminal, a current source time SrcTime_currand a current counted time STC_curr of the dedicated clock, in which both the counted time STC_curr and the source time SrcTime_curr are monotone increasing values;

calculating a difference between the counted time STC_curr and a second initial time STC_init to obtain a first relative time STC_diff; and calculating a difference between the source time SrcTime_curr and a third initial time SrcTime_init to obtain a second relative time SrcTime_diff; in which the second initial time STC_init is a counted time obtained when a first synchronization adjustment time is reached; and the third initial time SrcTime_initis a source time obtained when the first synchronization adjustment time is reached;

calculating a difference between the first relative time STC_diff and the second relative time SrcTime_diff to obtain a time error Time_errorbetween the dedicated clock and the audio or video source clock;

if the time error Time_error is larger than a preset maximum error threshold, then reducing an output frequency of the dedicated clock according to a preset first frequency refining step size; in which the preset maximum error threshold is larger than zero; and

if the time error Time_error is smaller than a preset minimum error threshold, then increasing the output frequency of the dedicated clock according to a preset second frequency refining step size; in which the preset minimum error threshold is smaller than zero.

Preferably, obtaining the current counted time of the dedicated clock includes:

obtaining a current clock counted value W_{ctr_curr}from the dedicated clock;

if currently a previous clock counted value W_{ctr_pre} is a preset initial value, then updating the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr};

if W_{ctr_curr}＜W_{ctr_pre}is satisfied, then adding a current number of rotations by one; in which an initial value of the number of rotations is 0;

calculating a clock accumulated count W_{ctr_accu}according to W_{ctr_accu}=W_{ctr_curr}+N×W_{ctr_max}; in which N is the current number of rotations; and updating the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr}; in which W_{ctr_max} is a maximum clock counted value of the dedicated clock; and

dividing the clock accumulated count W_{ctr_accu} by a current clock frequency of the dedicated clock to obtain the current counted time of the dedicated clock.

Embodiments of the present disclosure provide an audio or video playing apparatus, including: a dedicated clock, a rendering control unit and a clock synchronization unit; in which

the rendering control unit is configured to, in a process of audio or video playback, perform rendering synchronization control based on a representation timestamp of a to-be-rendered data frame and an elapsed playback time of a type of data corresponding to the to-be-rendered data frame, in which the elapsed playback time is obtained based on the dedicated clock of a playback terminal; and

the clock synchronization unit is configured to, in the process of audio or video playback, perform synchronization adjustment control on the dedicated clock based on a source time, so as to match the dedicated clock with the source time.

Preferably, the rendering control unit is configured to perform the rendering synchronization control, including:

Preferably, the rendering control unit is configured to obtain the elapsed playback time, including:

Preferably, the clock synchronization unit is configured to perform the synchronization adjustment control on the dedicated clock of the playback terminal, including:

Preferably, the apparatus further includes a dedicated timing unit;

in which the timing unit is configured to determine the current counted time of the dedicated clock according to a time obtaining instruction from the rendering control unit or the clock synchronization unit, and return the current counted time of the dedicated clock to the rendering control unit or the clock synchronization unit; in which determining the current counted time of the dedicated clock includes:

obtaining the current clock counted value W_{ctr_curr}from the dedicated clock;

Embodiments of the present disclosure further provide an audio or video playing device, including a processor and a memory;

in which the memory stores an application program executable by the processor, and configured to cause the processor to perform the audio or video playing method as mentioned above.

Embodiments of the present disclosure further provide a computer-readable storage medium, storing computer-readable instructions which are configured to perform the audio or video playing method as mentioned above.

As can be seen from the above technical schemes, according to the audio or video playing schemes provided by the embodiments of the present disclosure, in the audio or video playback process, the synchronization adjustment control is performed on the dedicated clock of the playback terminal based on the source time, which can ensure that the dedicated clock of the playback terminal matches the coding clock at the source in the audio or video playback process. Further in the rendering stage, the rendering synchronization control is performed based on the representation timestamp of the to-be-rendered data frame and the elapsed playback time obtained based on the dedicated clock of the playback terminal. In this way, on the one hand, using the dedicated clock which matches the source time achieves smooth playback, and on the other hand, performing synchronization in the rendering stage can meet the needs of audio or video playback with a high real-time requirement such as live broadcasting. In addition, performing the synchronization adjustment control on the dedicated clock of the playback terminal based on the source time can realize the decoupling of the clock at the source and the clock at the playback terminal. Further, the rendering stage has a clock precision requirement lower than a clock precision requirement for the coding clock at the source, which allows a tolerable error in the audio or video rendering, and therefore, the clock frequency at the playback terminal does not need to be as high as the source clock (e.g., up to 1GHz). With the decrease of the clock frequency, the power consumption is reduced too, and thus the implementation cost and power consumption of the dedicated clock at the playback terminal can be reduced.

FIG.1 is a schematic diagram of a flow of an audio or video playing method according to an embodiment of the present disclosure;

FIG.2 is a schematic diagram of a flow of a method for obtaining a counted time of a dedicated clock according to an embodiment of the present disclosure;

FIG.3 is a schematic diagram of a flow of a method for controlling synchronization adjustment of a dedicated clock at a playback terminal according to an embodiment of the present disclosure; and

FIG.4 is a diagram of a structure of an audio or video playing apparatus according to an embodiment of the present disclosure.

To make the objects, technical schemes and advantages of the present disclosure clearer, the present disclosure will be described in detail hereinafter with reference to accompanying drawings and detailed embodiments.

Consider that many programs broadcast by broadcast operators have a strong real-time property, and the conventional way of performing synchronization by means of buffering in a decoding stage on the one hand will lead to a long latency of the audio or video playback, and the real-time property of audio or video playback cannot be guaranteed, and on the other hand, the smoothness of audio or video playback cannot be guaranteed. For this reason, the present disclosure performs rendering control in a rendering stage of audio or video playback based on a dedicated clock of UE, and periodically performs synchronization adjustment on the dedicated clock based on source times of a source during the playback process, which enables the speed of playing audio or video at the receiver to be consistent with the speed of generating the audio or the video, so as to achieve the effect of smooth audio or video playback.

Fig. 1 is a schematic diagram of a flow of an audio or video playing method according to an embodiment of the present disclosure. As shown in Fig.1, the embodiment includes the following:

Step 101, in a process of audio or video playback, a playback terminal performs rendering synchronization control based on a representation timestamp of a to-be-rendered data frame and an elapsed playback time of a type of data corresponding to the to-be-rendered data frame. The elapsed playback time is obtained based on a dedicated clock of the playback terminal.

In this step, after the playback terminal starts the audio or video playback procedure, it performs rendering synchronization control on each data frame which enters into a rendering stage after being de-multiplexed and processed by an audio or video decoder, to achieve the effect of smooth playback.

Herein, compared to the traditional buffer synchronizing method in the decoding stage, the present disclosure adopts synchronization control in the rendering stage, which on the one hand can satisfy the real-time property of audio or video playback, and on the other hand, can achieve the effect of smooth audio or video playback.

The elapsed playback time is to indicate a period of time playing a corresponding type of data recorded based on the dedicated clock after a first audio frame or a first video frame arrives at the renderer of the playback terminal and is rendered.

Preferably, in a detailed implementation, the elapsed playback time may be obtained as follows:

When the to-be-rendered data frame arrives at the renderer, obtain a current counted time of the dedicated clock, in which counted times of the dedicated clock are monotone increasing values; and calculate a difference between the counted time and a first initial time to obtain the elapsed playback time.

The first initial time is a counted time obtained when the first data frame of the same type as the to-be-rendered data frame arrives at the renderer in the audio or video playback process. Thus, an initial value of the elapsed playback time is zero. Since the counted times of the dedicated clock are progressively increasing time values, the elapsed playback time is a time increasing from zero.

Preferably, in an implementation, to accurately obtain a counted time of the dedicated clock, as shown in Fig.2, the counted time may be obtained as follows:

Step a1, obtain a current clock counted value W_{ctr_curr}from the dedicated clock of the playback terminal.

It is to be specified that, in this step, the dedicated clock generates pulses through hardware and counts the pulses, and divides a counted value by a frequency value to obtain a counted time. Usually a register with a limited number of bits is used for the clock hardware to store clock counts. Assuming that the number of bits of the register is M, and the clock frequency is Clock_Freq, then the maximum time can be indicated is (2^M -1)/Clock_Freq. When the maximum value is reached, it will rotate back and restart from 0. For example, for a register of 33 bits and 90KHz, the maximum value is (2³³-1)/90000, approximating 95,443 seconds or 26.5 hours. In order to make the counted times of the dedicated clock monotonously increase, the above-mentioned rotating back situation needs to be handled when counting the times, that is, the number of rotations needs to be considered when counting the times.

Step a2, if currently a previous clock counted value W_{ctr_pre} is a preset initial value, then update the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr}.

This step is used to update the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr} when a counted time is obtained for the first time.

Specifically, the initial value of W_{ctr_pre}may be set by a person skilled in the art according to actual needs, e.g., zero, but not limited thereto.

Step a3, if W_{ctr_curr}＜W_{ctr_pre} is satisfied, then add a current number of rotations by one.

An initial value of the number of rotations is 0.

Herein, if W_{ctr_curr}＜W_{ctr_pre}, then it indicates that a count rotation has occurred currently, and the number of rotations should be added by one.

Step a4, calculate a clock accumulated count W_{ctr_accu}according to W_{ctr_accu}=W_{ctr_curr}+N×W_{ctr_max}.

N is the current number of rotations; and the previous clock counted value W_{ctr_pre} is updated to the current clock counted value W_{ctr_curr}; W_{ctr_max} is a maximum clock counted value of the dedicated clock.

Herein, when calculating the clock accumulated count W_{ctr_accu}, the current number of rotations is considered, and therefore it is guaranteed that the counted times of the dedicated clock are monotone increasing.

Step a5, divide the clock accumulated count W_{ctr_accu} by a current clock frequency of the dedicated clock to obtain the current counted time of the dedicated clock.

The presentation timestamp (PTS) of a data frame may be calculated using an existing method. The presentation timestamp Frame_pts of a data frame may be calculated according to Frame_pts=(n-1)×Frame_duration, where n is a number of an audio data frame or a number of a video data frame, and the number is for data frames of the same type, i.e., indicating an n^th audio data frame or an n^th video data frame. Using video frame data as an example, a first PTS of a first video frame is 0, and PTSs of subsequent video frames are accumulated using Frame_duration. For example, for a video of 60fps, with Frame_duration being 16.667ms, the Frame_pts of the first video frame is 0ms, the Frame_pts of the second video frame is 16.667ms, and the Frame_pts of the N^th frame is (N-1) ×16.667ms. Thus, the Frame_pts is a time value increasing from 0. In this way, by comparing the presentation timestamp Frame_pts and the elapsed playback time which also increases from zero, whether a time when a to-be-rendered data frame arrives at the render matches a playback speed can be known.

Specifically, in step 101, the presentation timestamp of the to-be-rendered data frame and the elapsed playback time of a type of data corresponding to the to-be-rendered data frame are compared, and the rendering synchronization control is performed according to a compared result.

To achieve a better smooth playback effect, in an implementation, the rendering synchronization control may be performed as follows:

when the to-be-rendered data frame arrives at the renderer, if Frame_pts≤STC_elapsed≤Frame_pts+D_max is satisfied, then perform rendering based on the current to-be-rendered data frame; where STC_elapsedis currently the elapsed playback time, Frame_pts is the representation timestamp of the current to-be-rendered data frame, and D_max is a preset maximum allowable delayed representation time; 0≤D_max≤Frame_duration; in which Frame_duration is a representation duration of a single data frame;

if STC_elapsed<Frame_pts is satisfied, then after waiting for a period of time △t, perform the rendering based on the current to-be-rendered data frame, in which △t=Frame_pts-STC_elapsed; and

if STC_elapsed>Frame_pts+ D_max is satisfied, then discard the current to-be-rendered data frame.

Using the above methods, the smooth playback of audio frame data and the smooth playback of video frame data can be realized respectively. If Frame_pts≤STC_elapsed≤Frame_pts+D_max is satisfied, it indicates that the current time meets the requirement of smooth playback, and that it is suitable to render the to-be-rendered data frame. Therefore, the to-be-rendered data frame is rendered immediately.

D_max satisfies 0≤D_max≤Frame_duration, and a suitable value of D_max may be configured by a person skilled in the art according to an actual smoothness requirement. Preferably, to smoothly present pictures and meanwhile reducing discarding data frames, D_max=Frame_duration may be configured.

If STC_elapsed<Frame_ptsis satisfied, then it indicates that the current to-be-rendered data frame arrives at the renderer earlier than the speed of smooth playback, and the rendering may be performed based on the current to-be-rendered data frame after waiting for a certain interval (i.e, Frame_pts-STC_elapsed), so as to achieve the effect of smooth playback.

If STC_elapsed>Frame_pts+D_max is satisfied, then it indicates that the current to-be-rendered data frame arrives at the renderer later than the speed of smooth playback. Thus, the current to-be-rendered data frame misses a playback time that corresponds to the smooth playback requirement, and the current to-be-rendered data frame should be discarded.

In the above method, Frame_pts≤STC_elapsed≤Frame_pts+D_max is currently timing suitable for performing immediate rendering, so that a certain tolerable error between the dedicated clock and the source clock is allowable, and it is unnecessary for the precision of the dedicated clock and the precision of the source clock to be identical, and the precision is up to milliseconds or above, i.e., the clock frequency is larger than or equal to 1KHz, and the smooth audio or video playback can be realized, so as to reduce the requirement for the precision of the dedicated clock, and further reduce the power consumption and the production cost of the dedicated clock, and improve the applicability of the method.

Step 102, In the process of audio or video playback, the playback terminal performs synchronization adjustment control on the dedicated clock based on a source time, so as to match the dedicated clock with the source time.

In this step, to guarantee the accuracy of performing rendering synchronization control based on the dedicated clock of the playback terminal in step 101, in the process of audio or video playback, the synchronization adjustment control is performed on the dedicated clock, to match the dedicated clock with the source time, which realizes matching between the dedicated clock and the coding clock at the audio or video source.

Herein, since the dedicated clock of the playback terminal is a reference clock synchronized in the rendering stage, and each data frame has presentation duration, thus, the playback times of data frames have a tolerable floating range. In this way, the dedicated clock of the playback terminal is unnecessary to have a precision requirement as high as that of the coding clock at the source, as long as a time error between the dedicated clock and the source clock does not affect the viewing experience of the human eye. Therefore, by using the dedicated clock of the playback terminal in the rendering process to perform synchronization adjustment, and to perform the synchronization adjustment control on the dedicated clock in the playback process, so that the dedicated clock matches the source clock, the playback terminal and the source clock can be decoupled, which in turn, can reduce the cost of realizing the dedicated clock of the playback terminal.

Preferably, to enable the dedicated clock to better synchronize with the source clock, in an implementation, as shown in Fig.3, the synchronization adjustment control on the dedicated clock of the playback terminal may be performed periodically as follows:

Step 1021, when a preset synchronization adjustment time is reached, the playback terminal obtains a current source time SrcTime_currand a current counted time STC_currof the dedicated clock.

Both the counted time STC_currand the source time SrcTime_currare monotone increasing values, so that relative times calculated respectively based on the two parameters in the subsequent steps are progressively increasing values too.

In this step, when each synchronization adjustment time is reached, a current source time SrcTime_curr and a current counted time of the dedicated clock need to be obtained at the same time, so as to determine a time difference between the current dedicated clock and the audio or video source clock, and further to determine whether to perform synchronization adjustment on the dedicated clock based on the time difference, so as to make the time of the dedicated clock match with the source time.

In implementation, a person skilled in the art may configure a synchronization adjustment time according to an actual periodical adjusting policy, for example, determining the synchronization adjustment time according to a preset synchronization adjustment period. For the synchronization adjustment period, if it is configured to be too long, then it is unable to meet the synchronization adjustment timeliness requirement, and if it is configured to be too short, then it will result in too much control cost. The period, e.g., 10s, may be adaptively configured by a person skilled in the art according to a policy that meets the synchronization adjustment timeliness requirement and that reduces the control cost as much as possible, in consideration of the speed of accumulation of the error time between the dedicated clock and the source clock. However, the present disclosure is not limited to this.

Herein, the current counted time of the dedicated clock may be obtained in the same way as that in step 101, which will not be elaborated herein.

It is to be specified that source times are periodically sent from the source to the playback terminal. In this way, the playback terminal may obtain the source times by parsing source signals.

Step 1022, calculate a difference between the counted time STC_currand a second initial time STC_initto obtain a first relative time STC_diff; and calculate a difference between the source time SrcTime_currand a third initial time SrcTime_initto obtain a second relative time SrcTime_diff.

The second initial time STC_initis a counted time obtained when a first synchronization adjustment time is reached; and the third initial time SrcTime_initis a source time obtained when the first synchronization adjustment time is reached.

In this step, the first relative time STC_diff indicates a time interval between the counted time of the dedicated clock currently obtained and a counted time obtained when the first synchronization adjustment time is reached. The second relative time SrcTime_diff indicates a time interval between the source time currently obtained and the source time obtained when the first synchronization adjustment time is reached. The difference between the first relative time and the second relative time can reflect whether the dedicated clock matches the source clock.

Step 1023, calculate a difference between the first relative time STC_diffand the second relative time SrcTime_diffto obtain a time error Time_errorbetween the dedicated clock and the audio or video source clock.

Herein, if the dedicated clock matches the source clock, then the difference between the first relative time and the second relative time (i.e., the time error Time_error) will be very small, or otherwise, the difference will be quite large.

Step 1024, if the time error Time_erroris larger than a preset maximum error threshold, then reduce an output frequency of the dedicated clock according to a preset first frequency refining step size; the preset maximum error threshold is larger than zero; and

if the time error Time_erroris smaller than a preset minimum error threshold, then increase the output frequency of the dedicated clock according to a preset second frequency refining step size; the preset minimum error threshold is smaller than zero.

Herein, if the time error Time_error is larger than the preset maximum error threshold, and the maximum error threshold is larger than zero, then it indicates that the dedicated clock is faster than the source clock. In this case, it is necessary to reduce the output frequency of the dedicated clock so as to slow down the time counting of the dedicated clock, so that the dedicated clock matches the source clock. If the time error Time_error is smaller than the preset minimum error threshold, and the minimum error threshold is smaller than zero, then it indicates that the dedicated clock is slower than the source clock. In this case, it is necessary to increase the output frequency of the dedicated clock so as to speed up the time counting of the dedicated clock, so that the dedicated clock matches the source clock. If the time error Time_error is between the minimum error threshold and the maximum error threshold, then it indicates that the dedicated clock matches the source clock, and it is unnecessary to adjust the frequency of the dedicated clock.

Specifically, the first frequency refining step size is used to control the magnitude of reduction in the output frequency of the dedicated clock each time, and the second frequency refining step size is used to control the magnitude of an increase in the output frequency of the dedicated clock each time. Specifically, a person skilled in the art may set the values of the first frequency refining step size and the second frequency refining step size according to the actual requirement of convergence speed of the adjusting.

A person skilled in the art may set the maximum error threshold and the minimum error threshold according to a matching degree requirement of the dedicated clock and the source clock in an implementation. Preferably, the absolute values of the maximum error threshold and the minimum error threshold are same, but the present disclosure is not limited thereto.

As can be seen from the above method embodiment, in the audio or video playing method, in the process of audio or video playback, performing the synchronization adjustment control on the dedicated clock of the playback terminal based on the source time, can ensure that the dedicated clock of the playback terminal matches the coding clock at the source, and in the rendering stage, the rendering synchronization control is performed based on the representation timestamp of the to-be-rendered data frame and the elapsed playback time of a type of data corresponding to the to-be-rendered data frame obtained based on the dedicated time of the playback terminal. In this way, on the one hand, using the dedicated clock matching the source time achieves smooth playback, and on the other hand, performing synchronization in the rendering stage can meet the needs of audio or video playback with a high real-time requirement such as live broadcasting. In addition, performing the synchronization adjustment control on the dedicated clock of the playback terminal based on the source time can realize the decoupling of the clock at the source and the clock at the playback terminal. Further, the rendering stage has a clock precision requirement lower than a clock precision requirement for the coding clock at the source, which allows a tolerable error in the audio or video rendering, and therefore, the clock frequency at the playback terminal does not need to be as high as the source clock, and thus the implementation cost and power consumption of the dedicated clock at the playback terminal can be reduced.

Corresponding to the audio or video playing method embodiments, the embodiments of the present disclosure further provide an audio or video playing apparatus. As shown in Fig.4, the apparatus at least includes: a rendering control unit 401, a dedicated clock 402, and a clock synchronization unit 403.

The rendering control unit 401 is configured to, in a process of audio or video playback, perform rendering synchronization control based on a representation timestamp of a to-be-rendered data frame and an elapsed playback time of a type of data corresponding to the to-be-rendered data frame, in which the elapsed playback time is obtained based on the dedicated clock of a playback terminal.

The clock synchronization unit 403 is configured to, in the process of audio or video playback, perform synchronization adjustment control on the dedicated clock based on a source time, so as to match the dedicated clock with the source time.

In an implementation, the rendering control unit 401 is configured to perform the rendering synchronization control, including:

when the to-be-rendered data frame arrives at a renderer, if Frame_pts≤STC_elapsed≤Frame_pts+D_max is satisfied, then performing rendering based on the current to-be-rendered data frame; where STC_elapsedis currently the elapsed playback time, Frame_pts is the representation timestamp of the current to-be-rendered data frame, and D_max is a preset maximum allowable delayed representation time; 0≤D_max≤Frame_duration; where Frame_duration is a representation duration of a single data frame;

if STC_elapsed<Frame_pts is satisfied, then after waiting for a period of time △t, performing the rendering based on the current to-be-rendered data frame, where △t=Frame_pts-STC_elapsed; and

In an implementation, the rendering control unit 401 is configured to obtain the elapsed playback time, including:

In an implementation, the clock synchronization unit 402 is configured to perform the synchronization adjustment control on the dedicated clock, including:

In an implementation, the apparatus further includes a dedicated timing unit 404.

The timing unit 404 is configured to determine the current counted time of the dedicated clock according to a time obtaining instruction from the rendering control unit or the clock synchronization unit, and return the current counted time of the dedicated clock to the rendering control unit or the clock synchronization unit; in which determining the current counted time of the dedicated clock includes:

obtaining the current clock counted value W_{ctr_curr}from the dedicated clock;

Corresponding to the audio or video playing method embodiments, the embodiments of the present disclosure further provide an audio or video playing device, including a processor and a memory;

the memory stores an application program executable by the processor, and configured to cause the processor to perform the audio or video playing method described above.

The memory may be specifically implemented as a variety of storage media such as electrically erasable programmable read-only memory (EEPROM), flash memory, and programmable read-only memory (PROM). The processor may be implemented to include one or more central processing units or one or more field programmable gate arrays, where the field programmable gate array(s) integrate(s) one or more central processing unit cores. Specifically, the central processing unit or central processing unit core may be implemented as a CPU or micro control unit (MCU).

It should be specified that not all steps and modules in the above-mentioned processes and structural diagrams are necessary, and some steps or modules can be omitted according to actual needs. The order of executing the respective steps is not fixed and can be adjusted as needed. The division of the respective modules is just to facilitate the description and is functional. In actual implementation, a module may be implemented by multiple modules, and the functions of multiple modules may also be implemented by a same module. These modules may be located in a same device, or may be located in different devices.

The hardware modules in each embodiment may be implemented in a mechanical way or an electronic way. For example, a hardware module may include specially designed permanent circuits or logic devices (e.g., dedicated processors, such as FPGAs or ASICs) to carry out specific operations. A hardware module may also include programmable logic devices or circuits temporarily configured by software (for example, including general-purpose processors or other programmable processors) to perform specific operations. Whether adopting the mechanical way or adopting the dedicated permanent circuits or adopting the temporarily configured circuits (e.g., configured by software) to implement the hardware module can be determined according to cost and time considerations.

The embodiment of the present disclosure further provides a computer-readable storage medium on which stores computer-readable instructions, and the computer-readable instructions are configured to perform the audio or video playing method mentioned above.

Specifically, a system or an apparatus equipped with a storage medium may be provided, and on the storage medium, software program code for realizing the function of any one of the above-mentioned embodiments is stored, and a computer (or a CPU or an MPU) of the system or the apparatus is able to read and execute the program code stored on the storage medium. In addition, an operating system or the like operating on the computer may also be used to carry out part or all of the actual operations through instructions based on the program code. It is also possible to write the program code read from the storage medium to a memory set in an expansion board inserted into the computer or to a memory set in an expansion unit connected to the computer, and then a CPU in the expansion board or the expansion unit may be enabled to perform part or all of the actual operations based on the instructions of the program code, so as to realize the function of any one of the above-mentioned embodiments.

Implementations of storage media used to provide the program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tape, non-transitory memory card and ROM. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

Herein, “schematic” means “serving as an example, an instance, or illustration”, and any illustration or embodiment described as “schematic” in the present disclosure should not be construed as a more preferred or advantageous technical solution. In order to make the drawings concise, the respective drawings only schematically show parts relative to the present disclosure, but do not represent the actual structure when implemented as a product. In addition, in order to make the drawings concise and easy to understand, in some drawings, for components with the same structure or function in some drawings, only one of them is schematically shown, or only one of them is indicated. Herein, “one” does not mean that the number of relevant parts of the present disclosure is limited to “only one”, and “one” does not mean to exclude the situation where the number of relevant parts of the present disclosure is “more than one”. Herein, “above”, “below”, “in front of”, “behind”, “left”, “right”, “inside”, “outside”, etc. are only used to indicate relative positions between related parts, but not used to limit the absolute positions of these relevant parts.

The above descriptions are merely preferred embodiments of the present disclosure, but are not intended to limit the protect scope of the present disclosure. Any modification, equivalent replacement, or improvement made within the spirit and principle of this application shall all fall within the protection scope of the present disclosure.

Claims

An audio or video playing method, characterized by comprising:

in a process of audio or video playback, performing, by a playback terminal, rendering synchronization control based on a representation timestamp of a to-be-rendered data frame and an elapsed playback time of a type of data corresponding to the to-be-rendered data frame, and performing synchronization adjustment control on a dedicated clock of the playback terminal based on a source time, so as to match the dedicated clock with the source time; wherein the elapsed playback time is obtained based on the dedicated clock.
The method of claim 1, characterized in that performing the rendering synchronization control comprises:

when the to-be-rendered data frame arrives at a renderer, if Frame_pts≤STC_elapsed≤Frame_pts+D_max is satisfied, then performing rendering based on the current to-be-rendered data frame; wherein STC_elapsedis currently the elapsed playback time, Frame_pts is the representation timestamp of the current to-be-rendered data frame, and D_max is a preset maximum allowable delayed representation time; 0≤D_max≤Frame_duration; wherein Frame_duration is a representation duration of a single data frame;

if STC_elapsed<Frame_pts is satisfied, then after waiting for a period of time △t, performing the rendering based on the current to-be-rendered data frame, wherein △t=Frame_pts-STC_elapsed; and

if STC_elapsed>Frame_pts+ D_max is satisfied, then discarding the current to-be-rendered data frame.
The method of claim 1, characterized in that obtaining the elapsed playback time comprises:

when the to-be-rendered data frame arrives at a renderer, obtaining a current counted time of the dedicated clock, wherein counted times of the dedicated clock are monotone increasing values; and

calculating a difference between the counted time and a first initial time to obtain the elapsed playback time; wherein the first initial time is a counted time obtained when a first data frame of the audio or video playback arrived at the renderer.
The method of claim 1, characterized in that performing the synchronization adjustment control on the dedicated clock of the playback terminal comprises:

when a preset synchronization adjustment time is reached, obtaining, by the playback terminal, a current source time SrcTime_currand a current counted time STC_curr of the dedicated clock, wherein both the counted time STC_curr and the source time SrcTime_curr are monotone increasing values;

calculating a difference between the counted time STC_curr and a second initial time STC_init to obtain a first relative time STC_diff; and calculating a difference between the source time SrcTime_curr and a third initial time SrcTime_init to obtain a second relative time SrcTime_diff; wherein the second initial time STC_init is a counted time obtained when a first synchronization adjustment time is reached; and the third initial time SrcTime_initis a source time obtained when the first synchronization adjustment time is reached;

calculating a difference between the first relative time STC_diff and the second relative time SrcTime_diff to obtain a time error Time_errorbetween the dedicated clock and the audio or video source clock;

if the time error Time_error is larger than a preset maximum error threshold, then reducing an output frequency of the dedicated clock according to a preset first frequency refining step size; wherein the preset maximum error threshold is larger than zero; and

if the time error Time_error is smaller than a preset minimum error threshold, then increasing the output frequency of the dedicated clock according to a preset second frequency refining step size; wherein the preset minimum error threshold is smaller than zero.
The method of claim 3, characterized in that obtaining the current counted time of the dedicated clock comprises:

obtaining a current clock counted value W_{ctr_curr}from the dedicated clock;

if currently a previous clock counted value W_{ctr_pre} is a preset initial value, then updating the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr};

if W_{ctr_curr}＜W_{ctr_pre}is satisfied, then adding a current number of rotations by one; wherein an initial value of the number of rotations is 0;

calculating a clock accumulated count W_{ctr_accu}according to W_{ctr_accu}=W_{ctr_curr}+N×W_{ctr_max}; wherein N is the current number of rotations; and updating the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr}; wherein W_{ctr_max} is a maximum clock counted value of the dedicated clock; and

dividing the clock accumulated count W_{ctr_accu} by a current clock frequency of the dedicated clock to obtain the current counted time of the dedicated clock.
An audio or video playing apparatus, characterized by comprising: a dedicated clock, a rendering control unit and a clock synchronization unit; wherein

the rendering control unit is configured to, in a process of audio or video playback, perform rendering synchronization control based on a representation timestamp of a to-be-rendered data frame and an elapsed playback time of a type of data corresponding to the to-be-rendered data frame, wherein the elapsed playback time is obtained based on the dedicated clock of a playback terminal; and

the clock synchronization unit is configured to, in the process of audio or video playback, perform synchronization adjustment control on the dedicated clock based on a source time, so as to match the dedicated clock with the source time.
The apparatus of claim 6, characterized in that the rendering control unit is configured to perform the rendering synchronization control, including:

when the to-be-rendered data frame arrives at a renderer, if Frame_pts≤STC_elapsed≤Frame_pts+D_max is satisfied, then performing rendering based on the current to-be-rendered data frame; wherein STC_elapsedis currently the elapsed playback time, Frame_pts is the representation timestamp of the current to-be-rendered data frame, and D_max is a preset maximum allowable delayed representation time; 0≤D_max≤Frame_duration; wherein Frame_duration is a representation duration of a single data frame;

if STC_elapsed<Frame_pts is satisfied, then after waiting for a period of time △t, performing the rendering based on the current to-be-rendered data frame, wherein △t=Frame_pts-STC_elapsed; and

if STC_elapsed>Frame_pts+ D_max is satisfied, then discarding the current to-be-rendered data frame.
The apparatus of claim 6, characterized in that the rendering control unit is configured to obtain the elapsed playback time, including:

when the to-be-rendered data frame arrives at a renderer, obtaining a current counted time of the dedicated clock, wherein counted times of the dedicated clock are monotone increasing values; and

calculating a difference between the counted time and a first initial time to obtain the elapsed playback time; wherein the first initial time is a counted time obtained when a first data frame of the audio or video playback arrived at the renderer.
The apparatus of claim 6, characterized in that the clock synchronization unit is configured to perform the synchronization adjustment control on the dedicated clock of the playback terminal, including:

when a preset synchronization adjustment time is reached, obtaining, by the playback terminal, a current source time SrcTime_currand a current counted time STC_curr of the dedicated clock, wherein both the counted time STC_curr and the source time SrcTime_curr are monotone increasing values;

calculating a difference between the counted time STC_curr and a second initial time STC_init to obtain a first relative time STC_diff; and calculating a difference between the source time SrcTime_curr and a third initial time SrcTime_init to obtain a second relative time SrcTime_diff; wherein the second initial time STC_init is a counted time obtained when a first synchronization adjustment time is reached; and the third initial time SrcTime_initis a source time obtained when the first synchronization adjustment time is reached;

calculating a difference between the first relative time STC_diff and the second relative time SrcTime_diff to obtain a time error Time_errorbetween the dedicated clock and the audio or video source clock;

if the time error Time_error is larger than a preset maximum error threshold, then reducing an output frequency of the dedicated clock according to a preset first frequency refining step size; wherein the preset maximum error threshold is larger than zero; and

if the time error Time_error is smaller than a preset minimum error threshold, then increasing the output frequency of the dedicated clock according to a preset second frequency refining step size; wherein the preset minimum error threshold is smaller than zero.
The apparatus of claim 8, characterized in that the apparatus further comprises a dedicated timing unit;

wherein the timing unit is configured to determine the current counted time of the dedicated clock according to a time obtaining instruction from the rendering control unit or the clock synchronization unit, and return the current counted time of the dedicated clock to the rendering control unit or the clock synchronization unit; wherein determining the current counted time of the dedicated clock comprises:

obtaining the current clock counted value W_{ctr_curr}from the dedicated clock;

if currently a previous clock counted value W_{ctr_pre} is a preset initial value, then updating the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr};

if W_{ctr_curr}＜W_{ctr_pre}is satisfied, then adding a current number of rotations by one; wherein an initial value of the number of rotations is 0;

calculating a clock accumulated count W_{ctr_accu}according to W_{ctr_accu}=W_{ctr_curr}+N×W_{ctr_max}; wherein N is the current number of rotations; and updating the previous clock counted value W_{ctr_pre} to the current clock counted value W_{ctr_curr}; wherein W_{ctr_max} is a maximum clock counted value of the dedicated clock; and

dividing the clock accumulated count W_{ctr_accu} by a current clock frequency of the dedicated clock to obtain the current counted time of the dedicated clock.
An audio or video playing device, characterized by comprising a processor and a memory;

wherein the memory stores an application program executable by the processor, and configured to cause the processor to perform the audio or video playing method in claim 1.
A computer-readable storage medium, characterized by storing computer-readable instructions which are configured to perform the audio or video playing method in claim 1.