CN116261000A

CN116261000A - Audio and video synchronization method and device in cloud conference and electronic equipment

Info

Publication number: CN116261000A
Application number: CN202310175659.XA
Authority: CN
Inventors: 马华文
Original assignee: G Net Cloud Service Co Ltd
Current assignee: G Net Cloud Service Co Ltd
Priority date: 2023-02-28
Filing date: 2023-02-28
Publication date: 2023-06-13

Abstract

The embodiment of the application discloses an audio and video synchronization method, a device and electronic equipment in a cloud conference, wherein the method comprises the following steps: acquiring a first program clock reference and configuring the first program clock reference as a reference clock; under the condition that audio and video are not synchronized in the current cloud conference, calculating a corresponding clock difference value according to a first preset mode; acquiring a first presentation time stamp of audio and a second presentation time stamp of video corresponding to the current unsynchronized condition; and calculating corresponding audio synchronization time stamps and video synchronization time stamps according to the reference clock, the clock difference value, the first presentation time stamp and the second presentation time stamp so as to synchronously play corresponding audio and video.

Description

Audio and video synchronization method and device in cloud conference and electronic equipment

Technical Field

The invention relates to the technical field of audio and video, in particular to an audio and video synchronization method, an audio and video synchronization device, a storage medium, electronic equipment and a computer program product in a cloud conference.

Background

With the rapid development of video cloud conferences and the diversification of video conference media, the time domain and region limitations are broken, and the video conferences can be rapidly carried out anytime and anywhere. In order to have better experience in a cloud conference, synchronization of audio and video needs to be realized when different types of devices enter the conference; if the phenomenon of fast and slow sound and image caused by asynchronous audio and video occurs, the participants can have poor conference experience.

The existing common schemes for real-time audio and video synchronization are divided into a main audio synchronization scheme and an RTCP synchronization scheme. The main audio synchronization scheme is that after receiving the time stamps of the audio and the video, the receiving end adopts the time stamp of the audio as a clock to calibrate the time stamp of the video so as to decode and play. At this time, the audio data play is continuous and normal, and the audio and video synchronization is realized by caching video frames. The problem with this solution is that the process of audio and video synchronization cannot be achieved in case of network delay.

In the RTCP scheme, an RTCP packet is used at a transmitting end to carry a clock of a transmitting device to a receiving end. The receiving end judges whether to decode and display the video and the audio at the moment according to the received audio and video time stamps and the clock. The scheme can realize audio and video synchronization, but because the RTCP packet is transmitted unreliably by using UDP, the audio and video can be directly lost due to the fact that the RTCP packet is not received or is late.

The existing real-time audio and video synchronization scheme in the conference does not well solve the problems of equipment clock difference, audio and video mismatch when the network is delayed and the like, so that the conference use experience of users is affected. Moreover, the prior art schemes use UDP for unreliable transport so that the out of sync problem can be affected by the network.

How to solve the problem of asynchronous audio and video caused by clock difference of different devices and network delay is a technical problem to be solved.

Disclosure of Invention

Based on this, it is necessary to provide a method, an apparatus, a storage medium, an electronic device and a computer program product for synchronizing audio and video in a cloud conference, aiming at the problem that the existing method cannot meet the requirement of asynchronous audio and video caused by different device clock differences and network delay.

In a first aspect, an embodiment of the present application provides a method for synchronizing audio and video in a cloud conference, where the method includes:

acquiring a first program clock reference and configuring the first program clock reference as a reference clock;

under the condition that audio and video are not synchronized in the current cloud conference, calculating a corresponding clock difference value according to a first preset mode;

acquiring a first presentation time stamp of audio and a second presentation time stamp of video corresponding to the current unsynchronized condition;

and calculating corresponding audio synchronization time stamps and video synchronization time stamps according to the reference clock, the clock difference value, the first presentation time stamp and the second presentation time stamp so as to synchronously play corresponding audio and video.

In one embodiment, the calculating the corresponding audio synchronization time stamp and video synchronization time stamp according to the reference clock, the clock difference, the first presentation time stamp and the second presentation time stamp includes:

adding the clock difference value with the first presentation time stamp, and calculating to obtain a corresponding audio synchronization time stamp; and/or the number of the groups of groups,

and adding the second presentation time stamp to the clock difference value, and calculating to obtain a corresponding video synchronization time stamp.

In one embodiment, the calculating the corresponding clock difference according to the first preset manner includes:

acquiring the reference time, the first presentation time stamp and a second program clock reference corresponding to the data transmission;

subtracting the first presentation time stamp from the reference time, and adding a corresponding second program clock reference when transmitting data, so as to calculate the clock difference value.

In one embodiment, the method further comprises:

and calculating an adjusted audio decoding time stamp and a video decoding time stamp according to the reference clock, the clock difference value, the first decoding time stamp of the audio which is not synchronously played and the second decoding time stamp of the video which is not synchronously played.

In one embodiment, the calculating the adjusted audio decoding time stamp and video decoding time stamp according to the reference clock, the clock difference value, the first decoding time stamp of the unsynchronized audio and the second decoding time stamp of the unsynchronized video includes:

adding the first decoding time stamp to the clock difference value, and calculating to obtain an adjusted audio decoding time stamp; and/or the number of the groups of groups,

and adding the second decoding time stamp to the clock difference value, and calculating to obtain an adjusted video decoding time stamp.

In one embodiment, the method further comprises:

and respectively carrying out clock synchronization processing on any one of the servers through the clock server so as to obtain the reference clock.

In one embodiment, the clock synchronization processing for any one of the plurality of servers by the clock server includes:

acquiring a clock of the clock server, and taking the clock as the reference clock;

the clock synchronization processing is sequentially carried out on the clock of any one of the servers according to the reference clock at intervals of preset time length, so that the clock corresponding to any one of the servers is consistent with the reference clock; wherein the plurality of servers comprise any one of a distribution server, a video server, an audio server and a service server.

In one embodiment, the method further comprises:

and responding to the detected clock deviation of the transmission stream packaging end, calculating a correction clock according to a second preset mode, and carrying out compensation processing on the packaging clock based on the correction clock.

In a second aspect, an embodiment of the present application provides an apparatus for audio and video synchronization in a cloud conference, where the apparatus includes:

the first acquisition module is used for acquiring a first program clock reference;

a configuration module for configuring the first program clock reference as a reference clock;

the first calculation module is used for calculating a corresponding clock difference value according to a first preset mode under the condition that the audio and video are not synchronized in the current cloud conference;

the second acquisition module is used for acquiring a first presentation time stamp of the corresponding audio and a second presentation time stamp of the video under the current unsynchronized condition;

and the second calculation module is used for calculating the corresponding audio synchronization time stamp and video synchronization time stamp according to the reference clock, the clock difference value, the first presentation time stamp and the second presentation time stamp so as to synchronously play the corresponding audio and video.

In a third aspect, an embodiment of the present application provides an electronic device, including:

A processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method steps described above.

In a third aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program for performing the above-described method steps.

In a fourth aspect, an embodiment of the present application provides an electronic device, including:

a processor;

a memory for storing the processor-executable instructions;

In a fifth aspect, embodiments of the present application provide a computer program product comprising a computer program which, when executed by a processor, implements the above-described method steps.

In the embodiment of the application, a first program clock reference is acquired, and the first program clock reference is configured as a reference clock; under the condition that audio and video are not synchronized in the current cloud conference, calculating a corresponding clock difference value according to a first preset mode; acquiring a first presentation time stamp of audio and a second presentation time stamp of video corresponding to the current unsynchronized condition; and calculating corresponding audio synchronization time stamps and video synchronization time stamps according to the reference clock, the clock difference value, the first presentation time stamp and the second presentation time stamp so as to synchronously play corresponding audio and video. According to the audio and video synchronization method in the cloud conference, due to the fact that the first program clock reference is introduced, the first program clock reference is configured as a reference clock; thus, different servers all have the same reference clock; in addition, due to the fact that the clock difference value is introduced, under the condition that the audio and the video are not synchronized in the current cloud conference, clock correction processing is conducted on the first presentation time stamp of the audio and the second presentation time stamp of the video according to the clock difference value and the reference clock, corresponding audio synchronization time stamp and video synchronization time stamp are calculated, and synchronous playing of the audio and the video is finally achieved.

Drawings

Exemplary embodiments of the present invention may be more fully understood by reference to the following drawings. The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the application, and not constitute a limitation of the invention. In the drawings, like reference numerals generally refer to like parts or steps.

Fig. 1 is a flowchart of a method for synchronizing audio and video in a cloud conference according to an exemplary embodiment of the present application;

fig. 2 is a flowchart of clock synchronization processing performed by an NIP server in a practical application scenario;

fig. 3a is a flowchart of an overall process of performing audio and video synchronization by using a PCR clock in TS packaging in an actual application scenario; fig. 3b is a schematic diagram of a flow (1) in a specific processing procedure of performing audio and video synchronization by adopting a PCR clock in TS packaging in an actual application scenario; fig. 3c is a schematic diagram of a flow (2) in a specific processing procedure of performing audio and video synchronization by adopting a PCR clock in TS packaging in an actual application scenario; fig. 3d is a schematic diagram of a flow (3) in a specific processing procedure of performing audio and video synchronization by adopting a PCR clock in TS packaging in an actual application scenario;

Fig. 4 is a flowchart of a TS packing clock correction process in an actual application scenario;

FIG. 5 is a flow chart of real-time statistics of system code rates of actual packaging in an actual application scenario;

fig. 6 is a schematic structural diagram of an audio/video synchronization device 600 in a cloud conference according to an exemplary embodiment of the present application;

FIG. 7 illustrates a schematic diagram of an electronic device provided in an exemplary embodiment of the present application;

fig. 8 shows a schematic diagram of a computer-readable medium according to an exemplary embodiment of the present application.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

It is noted that unless otherwise indicated, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs.

In addition, the terms "first" and "second" etc. are used to distinguish different objects and are not used to describe a particular order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.

The embodiments of the present application provide a method and an apparatus for synchronizing audio and video in a cloud conference, an electronic device, and a computer readable medium, and the following description is made with reference to the accompanying drawings.

Referring to fig. 1, a flowchart of an audio/video synchronization method in a cloud conference according to some embodiments of the present application is shown, where as shown in the drawing, the audio/video synchronization method in the cloud conference may include the following steps:

step S101: a first program clock reference is obtained and configured as a reference clock.

In a possible implementation manner, the method for synchronizing audio and video in a cloud conference provided by the embodiment of the application further includes the following steps:

And respectively carrying out clock synchronization processing on any one of the servers through the clock server so as to obtain a reference clock.

In an actual application scenario, the plurality of servers include any one of a distribution server, a video server, an audio server, and a service server. In addition to the above servers, other types of servers can be introduced according to the requirements of different application scenarios, and the types of the servers are not particularly limited.

In one possible implementation manner, clock synchronization processing is performed on any one of the plurality of servers through the clock server, respectively, including the following steps:

acquiring a clock of a clock server, and taking the clock as a reference clock;

the clock synchronization processing is sequentially carried out on the clock of any one of the servers according to the reference clock at intervals of preset time length so that the clock corresponding to any one of the servers is consistent with the reference clock; the plurality of servers comprise any one of a distribution server, a video server, an audio server and a service server.

In the synchronization method provided in the embodiment of the present application, the preset duration of the interval is not particularly limited, for example, in a certain application scenario, the preset duration of the interval is set to four hours every interval. And will not be described in detail herein.

In an actual application scene, the clock server can be an NTP server, and clock synchronization can be performed by using the NTP server, so that clocks of a plurality of servers in the system can be synchronized; in this way, audio and video synchronization can be ensured.

As shown in fig. 2, a flowchart of clock synchronization processing performed by the NIP server in an actual application scenario is shown.

The following description is made with respect to the clock synchronization processing flow performed by the NIP server in fig. 2:

step a1: on clock server E (192.168.0.100), install ntp service, require modification of ntp profile, modified as follows:

restrict 192.168.0.0 netmask 255.255.255.0 nomodify；

the above command is to allow devices within the subnet IP of 192.168.0.0-192.168.0.254 to clock synchronize and not be able to modify the clock server.

Step a2: on the distribution server a, the ntp service is installed and needs to use the crontab configuration to periodically execute the clock of the synchronization server E:

the commands employed to periodically execute the clocks of the synchronization server E using the crontab configuration are as follows:

。

through the command, the preset time length of the configured interval can be 4 hours, so that the clock server E is synchronized once every 4 hours, and the clock synchronization of the distribution server A and the clock server E is ensured.

Step a3: the video server B, the audio server C, and the service server D employ the similar configuration method employed in the above step a2, so that clocks of the above servers (the clock of the distribution server a, the clock of the video server B, the clock of the audio server C, and the clock of the service server D) can be synchronized with the clock server E.

Step S102: and under the condition that the audio and video are not synchronized in the current cloud conference, calculating a corresponding clock difference value according to a first preset mode.

In one possible implementation manner, the calculating the corresponding clock difference value according to the first preset manner includes the following steps:

acquiring a reference time, a first presentation time stamp and a second program clock reference corresponding to the data transmission time;

subtracting the first presentation time stamp from the reference time, and adding a corresponding second program clock reference when transmitting data, so as to calculate a clock difference value.

In a specific application scenario, the clock difference may be calculated as follows:

wherein->

For the clock difference, pcr is the reference time, dts is the first presentation time stamp, ++>

For the corresponding second program clock reference when the data is transmitted.

Step S103: and acquiring a first presentation time stamp of the corresponding audio and a second presentation time stamp of the video under the current unsynchronized condition.

In an actual application scene, when the condition that the audio and the video are not synchronized in the current cloud conference is monitored, a first presentation time stamp of the audio under the current condition and a second presentation time stamp of the corresponding video can be obtained in real time.

Step S104: and calculating corresponding audio synchronization time stamps and video synchronization time stamps according to the reference clock, the clock difference value, the first presentation time stamp and the second presentation time stamp so as to synchronously play corresponding audio and video.

In one possible implementation, calculating the corresponding audio synchronization timestamp and video synchronization timestamp according to the reference clock, the clock difference, the first presentation timestamp and the second presentation timestamp includes the steps of:

adding a first presentation time stamp to the clock difference value, and calculating to obtain a corresponding audio synchronization time stamp; and/or the number of the groups of groups,

and adding a second presentation time stamp to the clock difference value, and calculating to obtain a corresponding video synchronization time stamp.

In a specific application scenario, the corresponding audio synchronization timestamp and video synchronization timestamp may be calculated as follows.

Wherein->

For the clock difference, pts is the presentation time stamp corresponding to the audio or video,/for the audio or video>

For the corresponding adjusted presentation time stamp of the audio or video, namely: the audio synchronous time stamp of the audio or the video synchronous time stamp of the video, and the synchronous playing of the audio and the video is realized through the synchronous time stamp.

and calculating the adjusted audio decoding time stamp and the video decoding time stamp according to the reference clock, the clock difference value, the first decoding time stamp of the unsynchronized playing audio and the second decoding time stamp of the unsynchronized playing video.

In one possible implementation, the adjusted audio decoding time stamp and video decoding time stamp are calculated according to a reference clock, a clock difference value, a first decoding time stamp of the unsynchronized audio and a second decoding time stamp of the unsynchronized video, comprising the steps of:

adding a first decoding time stamp to the clock difference value, and calculating to obtain an adjusted audio decoding time stamp; and/or the number of the groups of groups,

and adding a second decoding time stamp to the clock difference value, and calculating to obtain an adjusted video decoding time stamp.

In a specific application scenario, the adjusted audio decoding time stamp and video decoding time stamp may be calculated as follows.

Wherein->

For the clock difference, dts is the first decoding time stamp of the audio or the second decoding time stamp of the video,/v>

For an adjusted audio decoding time stamp or an adjusted video decoding time stamp.

As shown in fig. 3a, a flowchart of an overall audio/video synchronization process performed by using a PCR (Program Clock Reference) clock for TS packaging in an actual application scene, fig. 3b, a schematic diagram of a process (1) in an audio/video synchronization process performed by using a PCR clock for TS packaging in an actual application scene, and fig. 3c, a schematic diagram of a process (2) in an audio/video synchronization process performed by using a PCR clock for TS packaging in an actual application scene; fig. 3d is a schematic diagram of a flow (3) in a specific audio/video synchronization process of a TS packet in an actual application scenario by using a PCR clock.

In a specific application scenario, a specific processing flow of performing audio and video synchronization by using a PCR clock in TS packaging is shown in fig. 3a to 3d, and the specific processing flow is as follows:

step b1: after the TS packaging thread is successfully created, the packaging parameter initialization is needed. The table information (PAT program association table/PMT program mapping table/SDT service description section) is initialized for the first time, blank packet data is generated, the number of packets required to be separated by PAT/PMT/SDT/PCR data is calculated according to the following company according to the system code rate and the table interval, and pcr_pkt (PCR packet data) is preset to be generated.

；/>

；

；

；

Wherein the pcr_time standard requires less than 50ms, at least 50ms requires the presence of a PCR packet.

、

、

Default 300ms is sent once.

Step b2: resetting

、

、

、

A counter for performing an initial value assignment when the initial value is zero, the initial value being +.>

Spacing. When TS packing procedure starts to beat +.>

These data can be quickly parsed and decoded for display at the decoding side. Wherein, it is to be noted that->

Firstly, packaging normal audio and video data, and then, performing PCR packaging; thus, the decoding end cannot empty the data, namely: after the audio and video data line is packed, the PCR data is sent out.

Step b3: and in a thread starting state, starting to execute thread processing, and firstly executing dormancy for 1ms and then releasing resources.

Step b4: judging whether the initial value of pcr is less than 0, if yes, executing step b5, otherwise, executing step b6.

Step b5: the pcr initialization is carried out, and the specific steps are as follows:

step b51: the loop traverses all the packed input channels, if step b52 is not completed, otherwise step b6 is executed.

Step b52: and taking out a frame of data from the input channel queue, if the frame of data can be taken out, executing the step b53, and otherwise, executing the step b51 to continue traversing.

Step b53: fetching the dts (decoding time stamp ) data of the current frame, determining whether the initial pcr is less than 0 or the current dts is less than the initial

If yes, step b54 is performed, otherwise step b51 is performed.

Step b54: judging whether dts is smaller than

2 times (minimum PCR fluctuation value), if yes, will be initially +.>

Set to 0, otherwise go to step b55, wherein, < ++>

。

Step b55: setting an initial value

The current channel id is assigned as index pointer and step b51 is performed.

Step b56: the 90K clock is converted to a 27M crystal oscillator clock,

step b6 is performed.

Step b6: and (c) performing clock correction processing, and initializing clocks t0 and t_last, wherein t0 is the clock at the start of packaging, t_last is the clock of the last packaging cycle, and continuing to execute the step b7.

Step b7: calculating the actual code rate of the system, if the actual code rate of the system is different from the last time, executing the step b8, otherwise executing the step b9.

Step b8: and (3) using the calculated actual system code rate real_sys_bit, assigning the actual system code rate real_sys_bit to last_sys_bit, recalculating pcr_interval, pat_interval, pmt_interval and sdt_interval according to the formula in the step b1, and recalculating the actual TS packing number from t0 to t_last according to the following mode. The purpose of this is to: the system code rate is prevented from becoming, and less packing or more packing is controlled, so that the phenomenon of empty packet insertion caused by insufficient code rate is avoided.

The formula used to recalculate the actual TS packet count from t0 to t_last is as follows:

。

step b9: the total number of packets at the current system code rate is calculated as follows, and the current clock t is assigned to t_last.

The formula adopted for calculating the total number of packets under the current system code rate is specifically as follows:

。

step b10: and (3) performing cyclic packaging processing, namely judging whether the packaged number pktNum is smaller than N, and performing cyclic packaging processing when the packaged number pktNum is smaller than N, and executing the step b11, otherwise, executing the step b1.

Step b11: calculating a current clock stc according to the current packed number in the following way, and executing step 12;

the formula used to calculate the current clock stc from the current packed number is as follows:

。

step b12: from the packed queue, pushing out the data, and the specific processing flow is as follows:

step b12-1: b12-2, circularly encoding all input channels, and if the step b is finished;

step b12-2: checking whether the queue has data, if so, executing step b12-3, otherwise executing step b12-1.

Step b12-3: setting a send_pcr according to the type, and if the type is judged to be video, setting a pcr_wave_range of the video; if the type is judged to be audio, aud _pcr_wave of the audio is set.

Step b12-4: a frame of data is acquired, dts is obtained from the frame, converted from a 27M clock to a 90K clock,

. And calculate +.>

The cut_tim_sum is a clock difference value corresponding to whether a transmission stream is cut off or an abnormal phenomenon occurs in a cloud conference.

Step b12-5: calculation of

If the dts_pcr is less than 0 or greater than 10s (so as to determine whether there is an anomaly in the transport stream in the cloud conference), step b12-6 is executed if yes, otherwise step b12-7 is executed.

Step b12-6: calculation of

And sets the transmission flag to 1, and performs step b12-8.

b12-7: and judging whether dts_pcr is smaller than or equal to send_pcr, if so, setting the sending mark as 1, and executing b12-8.

b12-8: and (3) taking out the data from the queue, calculating the pts of the frame adjustment and the dts of the frame adjustment, assigning the pts and the dts to the frame data, and executing the step b12-9.

；

。

b12-9: pushing the processed data to a sending module and sending the data. Step b13 is performed.

Step b13: TS packaging is carried out, and the specific processing flow is as follows:

step b13-1: judging whether pcr_count is greater than or equal to pcr_interval, if yes, executing step b13-2, otherwise executing step b13-4;

step b13-2: acquiring a clock, judging whether the pcr clock is packed with the es, and if so, executing the steps to acquire the pkt data preset by the es; otherwise, set up

And b13-3, performing the step of initializing.

Step b13-3: setting pcr into pkt package of pcr, and packaging and outputting by adopting the following formula

The counter is reset in the manner shown and steps b13-7 are performed.

Step b13-4: judging whether the pat_count is greater than or equal to the pat_interval, if so, outputting the pat packet, updating the cycle count bit in the pat packet,

and step b13-7 is performed, otherwise step b13-5 is performed.

Step b13-5: judging whether the pmt_count is greater than or equal to the pmt_interval, if so, outputting a pmt packet, updating a cycle count bit in the pmt packet,

and executing step b13-7, otherwise executing step b13-6.

Step b13-6: judging whether sdt _count is greater than or equal to sdt _interval, if so, outputting sdt packet, updating cycle count bit in sdt packet,

and executing step b13-7, otherwise executing step b13-6.

Step b13-7: sequentially adding 1 to the pcr_count counter, the pat_count counter, the pmt_count counter and the sdt _count counter respectively, and executing the step b10.

In the method for synchronizing audio and video in a cloud conference provided in the embodiment of the present application, the manner of calculating the correction clock according to the second preset manner has different forms,

for example, in the application scenario 1, the process of calculating the correction clock according to the second preset manner is specifically as follows:

judging whether dts_stc is greater than

If so, the correction clock is performed upward according to the following rules, and the specific calculation mode of the correction clock is as follows:

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

When in use;

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；/>

when the duration value is at

In the time-course of which the first and second contact surfaces,

。

for another example, in the application scenario 2, the process of calculating the correction clock according to the second preset manner is specifically as follows:

determining whether dts_stc is less than

If so, the correction clock is performed according to the following rule, and the specific calculation mode of the correction clock is as follows:

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

wherein the value of correct_step is 1000, corresponding to a 500ns clock.

The above two modes are merely examples, and the algorithm corresponding to the second preset mode may be adjusted according to the requirements of different application scenarios, so as to accurately calculate the correction clock, and perform compensation processing on the packing clock based on the correction clock.

The process of calculating the correction clock according to the second preset mode adopted for other different application scenarios is not described in detail.

As shown in fig. 4, a flowchart of the TS packing clock correction process in the actual application scenario is shown.

In order to avoid clock deviation possibly occurring in the running process of the equipment, clock correction in TS packaging is adopted, and compensation clock processing is carried out through correction, so that audio and video synchronization can be achieved more accurately.

As shown in fig. 4, the specific flow of the TS packing clock correction process in the actual application scenario is as follows:

step c1: and (3) time correction, namely judging whether current flow break occurs, if so, executing a step c14, and if not, executing a step c2.

Step c2: judging whether the video queue head has data, if so, executing the step c3, otherwise, executing the step c4.

Step c3: the video data frame is fetched and the frame type is set to video, step c6 is performed.

Step c4: judging whether the audio queue head has data, if so, executing step c5, otherwise, executing step c14.

Step c5: the audio data frame is fetched and the frame type is set to audio, step c6 is performed.

Step c6: pcr _ wave is set according to the data type, and if it is video,

otherwise, let(s)>

Step c7 is performed.

Step c7: if the data type is video, the data frame_tail is fetched from the video queue tail, otherwise, the data frame_tail is fetched from the audio queue tail, and step c8 is performed.

Step c8: fetching dts_first and dts_tail from head frame and tail frame_tail, and calculating queue time interval

Step c9 is performed.

Step c9: converted 90K clock

And calculate +.>

Meanwhile, an average dts is calculated according to the following formula, and step c10 is performed.

Wherein->

Is a clock value of 1 second for a 90K clock.

Step c10: judging whether the duration is greater than dts_average, if so, executing step c11, otherwise, executing step c12.

Step c11: judging whether dts_stc is greater than

If so, according to the following ruleThe correction clock is run up, otherwise step c14 is performed.

When the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

wherein the value of correct_step is 1000, corresponding to a 500ns clock.

Step c12: judging whether the duration is smaller than dts_average or not, if yes, executing step c13, otherwise, executing step c14.

Step c 13: determining whether dts_stc is less than

If so, the correction clock is run down according to the following rules, otherwise, step c 14 is performed.

When the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

In the time-course of which the first and second contact surfaces,

；

when the duration value is at

When (I)>

；

Wherein the value of correct_step is 1000, corresponding to a 500ns clock.

Step c 14: and (3) finishing correction calculation of the correction_tim, and compensating the packing clock t by using the clock t=t+correction_tim.

As shown in fig. 5, a flow chart of real-time statistics of the system code rate of the actual package in the actual application scenario is shown.

As shown in fig. 5, a specific processing flow of real-time statistics of the system code rate of the actual package in the actual application scenario is as follows:

step d1: and (3) cycling through all input channels, if not, executing a step d2, otherwise, executing a step d8.

Step d2: judging whether the channel is used, if so, executing the step d3, otherwise, executing the step d1.

Step d3: the es_rate accumulates the transmission residual data amount, the es_rate accumulates the data amount waiting to be transmitted, the channel queue number frame_cnt is calculated, and step d4 is executed.

Step d4: if the loop counter loop is smaller than the frame_cnt, step d5 is performed, otherwise step d7 is performed.

Step d5: and judging whether the data is fetched from the queue according to the loop value of the counter, if so, executing a step d6, otherwise, executing a step d7.

Step d 6: the frame data is obtained and accumulated into the es_rate, the counter loop is incremented by 1, and step d4 is executed.

Step d7: d1, calculating a code rate of a current channel according to the following steps, and executing the step d1;

。

step d8: the actual system code rate is calculated as follows:

。

step d9: if real_sys_bit is less than the sys_bit_rate, then,

wherein the sys_bit_rate is a system code rate value set by the package, and step d10 is performed.

Step d10: if real_sys_bit is greater than 10000000, then,

i.e. the maximum system code rate 10M, step d11 is performed.

Step d11: the real_sys_bit is returned to facilitate the packing process.

According to the audio and video synchronization method in the cloud conference, clock synchronization of the NTP server can ensure that clocks of devices used in a system are kept consistent; further, TS is packed and sent out with a PCR clock, so that decoding display output is judged at a receiving end according to the PCR, the audio DTS and the video DTS, and audio and video synchronization is ensured; in addition, through the clock correction mechanism, under the condition that the audio and video are not synchronous due to clock deviation in the running process of the equipment, the time lost in certain processing processes can be compensated through the clock correction mechanism, so that the clock deviation phenomenon possibly occurring when the TS packaging end is used for packaging is avoided, synchronous playing of the audio and the video is realized, and visual experience of users in the cloud conference is improved.

According to the audio and video synchronization method in the cloud conference, due to the fact that the first program clock reference is introduced, the first program clock reference is configured as a reference clock; thus, different servers all have the same reference clock; in addition, due to the fact that the clock difference value is introduced, under the condition that the audio and the video are not synchronized in the current cloud conference, clock correction processing is conducted on the first presentation time stamp of the audio and the second presentation time stamp of the video according to the clock difference value and the reference clock, corresponding audio synchronization time stamp and video synchronization time stamp are calculated, and synchronous playing of the audio and the video is finally achieved.

In the foregoing embodiment, a method for synchronizing audio and video in a cloud conference is provided, and corresponding to the method, the application also provides a device for synchronizing audio and video in a cloud conference. The audio and video synchronization device in the cloud conference can implement the audio and video synchronization method in the cloud conference, and the audio and video synchronization device in the cloud conference can be realized in a mode of software, hardware or combination of software and hardware. For example, the audio/video synchronization device in the cloud conference may include integrated or separated functional modules or units to perform corresponding steps in the above methods.

Fig. 6 is a schematic diagram of an audio/video synchronization device in a cloud conference according to some embodiments of the present application. Since the apparatus embodiments are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points. The device embodiments described below are merely illustrative.

As shown in fig. 6, the audio/video synchronization device 600 in the cloud conference may include:

a first obtaining module 601, configured to obtain a first program clock reference;

a configuration module 602 for configuring a first program clock reference as a reference clock;

the first calculating module 603 is configured to calculate a corresponding clock difference according to a first preset manner when the audio and video in the current cloud conference are not synchronized;

a second obtaining module 604, configured to obtain a first presentation timestamp of the audio and a second presentation timestamp of the video corresponding to the current unsynchronized situation;

the second calculating module 605 is configured to calculate a corresponding audio synchronization timestamp and a corresponding video synchronization timestamp according to the reference clock, the clock difference, the first presentation timestamp and the second presentation timestamp, so as to perform synchronous playing of the corresponding audio and video.

In some implementations of the embodiments of the present application, the second computing module 605 is specifically configured to:

In some implementations of the embodiments of the present application, the first computing module 603 is specifically configured to:

In some implementations of the embodiments of the present application, the apparatus 600 may further include:

a third calculation module (not shown in fig. 6) for calculating the adjusted audio decoding time stamp and video decoding time stamp according to the reference clock, the clock difference value, the first decoding time stamp of the unsynchronized audio and the second decoding time stamp of the unsynchronized video.

In some implementations of the embodiments of the present application, the third computing module is specifically configured to:

a synchronization module (not shown in fig. 6) for performing clock synchronization processing on any one of the plurality of servers through the clock server, respectively, to obtain a reference clock.

In some implementations of the embodiments of the present application, the synchronization module is specifically configured to:

acquiring a clock of a clock server, and taking the clock as a reference clock;

a fourth calculating module (not shown in fig. 6) for calculating a correction clock according to a second preset manner in response to detecting that the clock of the transport stream packing end is deviated;

A compensation module (not shown in fig. 6) for performing compensation processing on the package clock based on the correction clock.

In some implementations of the embodiments of the present application, the synchronization device 600 for audio and video in a cloud conference provided by the embodiments of the present application has the same beneficial effects as the synchronization method for audio and video in a cloud conference provided by the foregoing embodiments of the present application due to the same inventive concept.

The embodiment of the application also provides an electronic device corresponding to the audio/video synchronization method in the cloud conference provided by the foregoing embodiment, where the electronic device may be an electronic device for a server, for example, a server, including an independent server, a distributed server cluster, and the like, so as to execute the audio/video synchronization method in the cloud conference; the electronic device may also be an electronic device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., so as to execute the above method for synchronizing audio and video in the cloud conference.

Referring to fig. 7, a schematic diagram of an electronic device according to some embodiments of the present application is shown. As shown in fig. 7, the electronic device 40 includes: processor 400, memory 401, bus 402 and communication interface 403, processor 400, communication interface 403 and memory 401 being connected by bus 402; the memory 401 stores a computer program that can be run on the processor 400, and when the processor 400 runs the computer program, the method for synchronizing audio and video in the cloud conference described in the application is executed.

The memory 401 may include a high-speed random access memory (RAM: random Access Memory), and may further include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. The communication connection between the system network element and at least one other network element is implemented via at least one communication interface 403 (which may be wired or wireless), the internet, a wide area network, a local network, a metropolitan area network, etc. may be used.

Bus 402 may be an ISA bus, a PCI bus, an EISA bus, or the like. The buses may be divided into address buses, data buses, control buses, etc. The memory 401 is configured to store a program, and the processor 400 executes the program after receiving an execution instruction, and the method for synchronizing audio and video in a cloud conference disclosed in any embodiment of the present application may be applied to the processor 400 or implemented by the processor 400.

The processor 400 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in the processor 400 or by instructions in the form of software. The processor 400 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401, and in combination with its hardware, performs the steps of the above method.

The electronic equipment provided by the embodiment of the application and the method for synchronizing the audio and video in the cloud conference provided by the embodiment of the application have the same beneficial effects as the method adopted, operated or realized by the electronic equipment and the method for synchronizing the audio and video in the cloud conference.

The embodiment of the present application further provides a computer readable medium corresponding to the method for synchronizing audio and video in a cloud conference provided in the foregoing embodiment, referring to fig. 8, the computer readable storage medium is shown as an optical disc 50, on which a computer program (i.e. a program product) is stored, where the computer program when executed by a processor performs the method for synchronizing audio and video in a cloud conference.

It should be noted that examples of the computer readable storage medium may also include, but are not limited to, a phase change memory (PRAM), a Static Random Access Memory (SRAM), a Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a flash memory, or other optical or magnetic storage medium, which will not be described in detail herein.

The computer readable storage medium provided by the above embodiment of the present application and the method for synchronizing audio and video in a cloud conference provided by the embodiment of the present application are the same inventive concept, and have the same beneficial effects as the method adopted, operated or implemented by the application program stored therein.

It is noted that the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the embodiments, and are intended to be included within the scope of the claims and description.

Claims

1. A synchronization method of audio and video in a cloud conference comprises the following steps:

2. The method of claim 1, wherein the calculating corresponding audio and video synchronization time stamps from the reference clock, the clock difference, the first presentation time stamp, and the second presentation time stamp comprises:

3. The method according to claim 1 or 2, wherein the calculating the corresponding clock difference according to the first preset manner comprises:

4. A method according to claim 3, further comprising:

5. The method of claim 4, wherein the calculating adjusted audio decoding time stamp and video decoding time stamp from the reference clock, the clock difference, a first decoding time stamp of the unsynchronized playback audio, and a second decoding time stamp of the unsynchronized playback video comprises:

6. The method of claim 1, further comprising:

7. The method of claim 6, wherein the clock synchronization processing for any one of the plurality of servers by the clock server includes:

8. The method of claim 1, further comprising:

9. An audio and video synchronization device in a cloud conference, comprising:

10. An electronic device, the electronic device comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any one of the preceding claims 1 to 8.