WO2020248649A1

WO2020248649A1 - Audio and video data synchronous playback method, apparatus and system, electronic device and medium

Info

Publication number: WO2020248649A1
Application number: PCT/CN2020/081104
Authority: WO
Inventors: 许赫赫
Original assignee: 北京大米科技有限公司
Priority date: 2019-06-13
Filing date: 2020-03-25
Publication date: 2020-12-17
Also published as: CN110234028A

Abstract

The present disclosure relates to an audio and video data synchronous playback method, apparatus and system, an electronic device and a computer readable medium. Said method comprises: acquiring a first audio and video data packet from a first user terminal; generating a second audio and video data packet by means of the first audio and video data packet and timestamp data; and transmitting, by means of a content delivery network, the second audio and video data packet to a second user terminal for playback. The audio and video data synchronous playback method, apparatus and system, the electronic device and the computer readable medium of the present disclosure can solve the problem that audio and video data and signaling information sent by a user are not synchronous in the process of live broadcast of videos.

Description

Method, device, system, electronic equipment and medium for synchronous playback of audio and video data

Technical field

The present disclosure relates to the field of computer information processing, and in particular, to a method, device, system, electronic equipment, and computer-readable medium for synchronous playback of audio and video data.

Background technique

With the development of transmission technology and the increasing demand for real-time communication among users, real-time video transmission technologies have emerged. For example, video live broadcast technology, the basic processing steps of current video live broadcast technology are: live broadcast terminals usually generate two types of data: audio and video data and signaling data. Among them, the collected and encoded audio and video data will pass CDN (Content Delivery Network). , The content distribution network) is transmitted to other terminals, and the signaling data is directly transmitted to other terminals, so that other users can watch live broadcast or conduct video communication.

Since most of the audio and video data is transmitted to the CDN using the RTMP protocol (Real-Time Messaging Protocol), and then distributed to other terminals by the CDN, the signaling information is directly transmitted to other terminals through the TCP protocol. Therefore, the two transmission methods may cause the problem of out of synchronization. For users, asynchronous audio, video and signaling will cause great troubles. In the current technical solutions, none of the above-mentioned problems have been solved. This technical dilemma has brought great inconvenience to users.

Therefore, there is a need for a new method, device, system, electronic equipment, and computer-readable medium for synchronous playback of audio and video data.

The above-mentioned information disclosed in the background section is only used to enhance the understanding of the background of the present disclosure, so it may include information that does not constitute the prior art known to those of ordinary skill in the art.

Summary of the invention

In view of this, the present disclosure provides a method, device, system, electronic equipment, and computer readable medium for synchronous playback of audio and video data, which can solve the problem that audio and video data is not synchronized with the signaling information sent by the user during the video live broadcast process. .

Other characteristics and advantages of the present disclosure will become apparent through the following detailed description, or partly learned through the practice of the present disclosure.

According to an aspect of the present disclosure, a method for synchronously playing audio and video data is provided. The method includes: acquiring a first audio and video data packet from a first user terminal; and generating a second audio and video data packet from the first audio and video data packet and time stamp data. Audio and video data packets; and transmitting the second audio and video data packets to a second user terminal through a content distribution network for playback.

In an exemplary embodiment of the present disclosure, generating the second audio and video data packet from the first audio and video data packet and the time stamp data includes: converting the first audio and video data packet to generate the target audio and video data Data packet; and generating the second audio and video data packet through the target audio and video data packet and the time stamp data.

In an exemplary embodiment of the present disclosure, generating the second audio and video data packet by using the target audio and video data packet and the time stamp data includes: generating the time stamp data by the universal standard time; The time stamp data generates supplementary enhancement information; and the second audio and video data packet is generated through the target audio and video data packet and the supplementary enhancement information.

In an exemplary embodiment of the present disclosure, generating the second audio and video data packet by using the target audio and video data packet and the time stamp data includes: generating the time stamp data by universal standard time; and The time stamp data is written into the file name of the target audio and video data packet to generate the second audio and video data packet.

In an exemplary embodiment of the present disclosure, converting the first audio and video data packet to generate the target audio and video data packet includes: converting the first audio and video data packet based on real-time communication technology to generate Target audio and video data packets based on real-time messaging protocol.

In an exemplary embodiment of the present disclosure, performing conversion processing on the first audio and video data packet to generate a target audio and video data packet includes: performing format conversion processing on the data format of the first audio and video data packet to Generating the target audio and video data packet; and/or performing protocol conversion processing on the communication protocol of the first audio and video data packet to generate the target audio and video data packet; and/or converting the first audio and video data The packet transmission protocol performs protocol conversion processing to generate the target audio and video data packet.

In an exemplary embodiment of the present disclosure, performing format conversion processing on the data format of the first audio and video data packet includes: performing format conversion processing on the video format in the first audio and video data packet; and/ Or perform format conversion processing on the audio format in the first audio and video data packet.

According to one aspect of the present disclosure, a method for synchronously playing audio and video data is provided. The method includes: obtaining at least one second audio and video data packet and at least one signaling information; and obtaining first time stamp data by parsing the signaling information Obtain second time stamp data by parsing the second audio and video data packet; and combine the at least one signaling information with the at least one second time stamp based on the first time stamp data and the second time stamp data Audio and video data packets are sorted in time for sequential playback.

In an exemplary embodiment of the present disclosure, the at least one signaling information and the at least one second audio and video data packet are processed based on the first time stamp data and the second time stamp data. The time sorting includes: generating a play queue of the at least one second audio and video data packet based on the play time axis determined by the second time stamp data; and sorting the at least one message based on the first time stamp data Let information be inserted into the play queue.

In an exemplary embodiment of the present disclosure, the inserting the at least one signaling information into the play queue based on the first time stamp data includes: generating an exhibition time through the first time stamp data; The display time is compared with the play time in the play time axis, and the at least one signaling information is inserted into the play queue according to the comparison result.

According to one aspect of the present disclosure, a method for synchronously playing audio and video data is proposed. The method includes: acquiring audio and video data, where the audio and video data includes at least one first audio and video data packet and at least one signaling information; and web-based The instant messaging protocol sends the at least one first audio and video data packet to the at least one second client through the relay server, and sends the at least one signaling information to the at least one second client based on the transmission control protocol, To play audio and video data.

According to one aspect of the present disclosure, a device for synchronously playing audio and video data is provided. The device includes: a data transfer module for obtaining a first audio and video data packet from a first user terminal; and an information adding module for passing the An audio and video data packet and time stamp data generate a second audio and video data packet; and a data transmission module for transmitting the second audio and video data packet to a second user terminal through a content distribution network for playback. .

According to one aspect of the present disclosure, a device for synchronously playing audio and video data is provided. The device includes: a data receiving module for acquiring at least one second audio and video data packet and at least one signaling information; a first time module for The first time stamp data is obtained by analyzing the signaling information; the second time module is used to obtain the second time stamp data by analyzing the second audio and video data packet; and the time sorting module is used to obtain the second time stamp data based on the first The time stamp data and the second time stamp data sort the at least one signaling information and the at least one second audio and video data packet in time so as to be played in sequence.

According to one aspect of the present disclosure, a device for synchronous playback of audio and video data is provided. The device includes: a data acquisition module for acquiring audio and video data. The audio and video data includes at least one first audio and video data packet and at least one message. Command information; and a data sending module, configured to send the at least one first audio and video data packet to at least one second client through the relay server based on the web instant messaging protocol, and to send the at least one signaling based on the transmission control protocol The information is sent to the at least one second user terminal for audio and video data playback.

According to one aspect of the present disclosure, a system for synchronous playback of audio and video data is provided. The system includes: a first user terminal for acquiring audio and video data, the audio and video data including at least one first audio and video data packet and at least one Signaling information; and sending the at least one first audio and video data packet to at least one second client through the relay server based on the web instant messaging protocol, and sending the at least one signaling information to the at least one second user terminal based on the transmission control protocol At least one second client terminal for playing audio and video data; a relay server for acquiring a first audio and video data packet from the first client terminal; generating a second audio and video data packet from the first audio and video data packet and time stamp data A data packet; and transmitting the second audio and video data packet through a content distribution network to a second user terminal for playback; and a content distribution network server for transmitting the second video data packet to the second user And a second user terminal for obtaining at least one second audio and video data packet and at least one signaling information; obtaining first time stamp data by analyzing the signaling information; and analyzing the second audio and video data packet Acquiring second time stamp data; and chronologically sorting the at least one signaling information and the at least one second audio and video data packet based on the first time stamp data and the second time stamp data, so as to be sequentially Play.

According to an aspect of the present disclosure, an electronic device is provided. The electronic device includes: one or more processors; a storage device for storing one or more programs; when one or more programs are used by one or more processors Execution enables one or more processors to implement the above method.

According to an aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and the program is executed by a processor to implement the above method.

According to the method, device, system, electronic device and computer-readable medium for synchronous playback of audio and video data of the present disclosure, a second audio and video data packet is generated from the first audio and video data packet and time stamp data; and The audio and video data packets are transmitted to the second user terminal through the content distribution network for display, which can solve the problem that the audio and video data is not synchronized with the signaling information sent by the user during the live video broadcast process.

It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present disclosure.

Description of the drawings

By describing its exemplary embodiments in detail with reference to the accompanying drawings, the above and other objectives, features, and advantages of the present disclosure will become more apparent. The drawings described below are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

Fig. 1 is a block diagram showing a system for synchronously playing audio and video data according to an exemplary embodiment.

Fig. 2 is a schematic diagram showing an application scenario of a system for synchronously playing audio and video data according to an exemplary embodiment.

Fig. 3 is a flowchart showing a method for synchronously playing audio and video data according to an exemplary embodiment.

Fig. 4 is a flow chart showing a method for synchronously playing audio and video data according to another exemplary embodiment.

Fig. 5 is a flowchart showing a method for synchronously playing audio and video data according to another exemplary embodiment.

Fig. 6 is a flowchart showing a system for synchronously playing audio and video data according to another exemplary embodiment.

Fig. 7 is a block diagram showing a device for synchronously playing audio and video data according to an exemplary embodiment.

Fig. 8 is a block diagram showing a device for synchronously playing audio and video data according to another exemplary embodiment.

Fig. 9 is a block diagram showing a device for synchronously playing audio and video data according to another exemplary embodiment.

Fig. 10 is a block diagram showing an electronic device according to an exemplary embodiment.

Fig. 11 is a block diagram showing a computer-readable medium according to an exemplary embodiment.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided so that this disclosure will be comprehensive and complete, and fully convey the concept of the example embodiments To those skilled in the art. In the figures, the same reference numerals denote the same or similar parts, and thus their repeated description will be omitted.

Furthermore, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

The block diagrams shown in the drawings are merely functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices. entity.

The flowchart shown in the drawings is only an exemplary description, and does not necessarily include all contents and operations/steps, nor does it have to be performed in the described order. For example, some operations/steps can be decomposed, and some operations/steps can be combined or partially combined, so the actual execution order may be changed according to actual conditions.

It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teaching of the concepts of the present disclosure. As used herein, the term "and/or" includes any one and all combinations of one or more of the associated listed items.

Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of exemplary embodiments, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present disclosure, and therefore cannot be used to limit the protection scope of the present disclosure.

The inventor of the present disclosure found that in the process of online teaching on the network, the student end and the teacher end can use their respective terminals to implement the process of remote video teaching through the Internet. Among them, for any client, the image and sound of the other party it receives are audio and video data, and the text, red envelopes and other information it receives are signaling information. For example, when the host said in the live broadcast: I want to send a red envelope, and immediately sent the red envelope. However, because the audio and video data is not synchronized with the signaling information, the user may first receive the red packet information sent by the teacher, and then receive the teacher's audio data.

In the prior art, the current CDN manufacturers usually only support the method of transmitting audio and video data using the RTMP (Real Time Messaging Protocol) protocol. Therefore, in the existing video live broadcast technology, the anchor terminal uses the RTMP protocol to transmit real-time audio and video data to the CDN, and then the CDN distributes the real-time audio and video data to each user terminal, and then plays the real-time video data in the user terminal For live video streaming.

This method of using RTMP protocol to transmit audio and video data in the host terminal requires a large amount of operating resources of the host terminal, which makes the host terminal data processing pressure is huge, and the delay phenomenon often occurs. This phenomenon not only takes up the higher system operation Ability also creates the problem of asynchrony between audio and video data and signaling information. In order to solve the problems in the prior art, the present disclosure proposes a method for synchronously playing audio and video data. The method in the present disclosure will be described in detail below through specific embodiments.

It is worth mentioning that in the embodiments of the present disclosure, the application scenario of live video broadcasting will be taken as an example to describe the method and device implementation of the synchronous playback of audio and video data, but it is worth mentioning that The method and device for synchronous playback of audio and video data can be applied to other actual scenes that require audio and video data transmission, and the non-disclosure is not limited to this.

As shown in FIG. 1, the application scenario may include a first client device 101, a relay server 102, a content distribution network server 103, a network 104, and a second client device 105. The network 104 is used to provide the medium of the communication link between the first client device 101 and the transfer server 102; the network 104 is also used to provide the medium of the communication link between the transfer server 102 and the content distribution network server 103; the network 104 It is also used as a medium for providing a communication link between the content distribution network server 103 and the second client device 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

Among them, the first client device 101 may be various electronic devices that have a display screen and a camera and support data transmission, and the second client device 105 may be various electronic devices that have a display screen and support video playback. The device 101 and the second client device 105 may include, but are not limited to, a smart phone, a tablet computer, a laptop portable computer, a desktop computer, and so on.

Among them, the first client device 101 and the second client device 105 can be installed with various communication client applications, such as live video platform applications, web browser applications, search applications, instant messaging tools, email clients, Social platform software, etc.

In a live broadcast application scenario, the host user can use the first client device 101 to interact with the relay server 102 via the network 104 to receive or send audio and video data, etc. The host user can also use the first client device 101 to interact with the content distribution network server 103 and the second client device 105 through the network 104 to receive or send audio and video data or various other signaling information; content distribution network server 103 can interact with the second client device 105 via the network 104.

In the live broadcast application scenario, the host user can use the first client device 101 to interact with the content distribution network server 103 and multiple second client devices 105 through the network 104 to receive or send audio and video data or various other Data, etc.; the content distribution network server 103 can interact with multiple second client devices 105 via the network 104.

Among them, the first client device 101 may, for example, obtain audio and video data, the audio and video data including at least one first audio and video data packet and at least one signaling information; The at least one first audio and video data packet is sent to at least one second user terminal 105 through the transit server 102, and the at least one signaling information is sent to the at least one second user terminal 105 based on the transmission control protocol to Play audio and video data.

The relay server 102 may be a server that provides various services, for example, a data processing server that provides support for the audio and video data sent by the anchor user using the first client device 101. The relay server 102 may analyze and process the received audio and video data, and send the processing result (for example, audio and video data with time stamp data) to the content distribution network server 103.

The relay server 102 can, for example, obtain the first audio and video data packet from the first client device 101; the relay server 102 can, for example, generate the second audio and video data packet from the first audio and video data packet and the time stamp data; the relay server 102 For example, the second audio and video data packet may be transmitted to the second client device 105 through a content distribution network for playback.

Furthermore, the transit server 102 may be a physical server, or may be composed of multiple servers, for example.

The content distribution network server 103 may be a server that provides data transmission, for example, a data transmission server that provides support for audio and video data sent by the relay server 102. The content distribution network server 103 may process the received audio and video data before forwarding, and send the processing result (for example, audio and video data in a predetermined format) to the second content distribution client device 105.

Furthermore, the content distribution network server 103 may be a physical server, or may be composed of multiple servers, for example. The content distribution network server 103 is a part of the CDN network. The basic idea of the CND network is to avoid bottlenecks and links on the Internet that may affect the data transmission speed and stability as much as possible, so as to make content transmission faster and more stable. By placing content distribution network servers 103 everywhere in the CDN network constituted by a layer of intelligent virtual network on the basis of the existing Internet, the CDN composed of multiple content distribution network servers 103 can real-time according to network traffic and transit servers 102. Comprehensive information such as the connection of the second client device 105, the load status, the distance to the second client device 105 and the response time, etc. redirect the request of the non-host user (ordinary user) to the service node closest to the ordinary user. The purpose is to enable ordinary users in the present disclosure to obtain audio and video data nearby, solve the congestion of the Internet, and improve the response speed of the second client device 105 receiving audio and video data.

Furthermore, to put it simply, the CND network is a strategically deployed overall system, including four requirements for distributed storage, load balancing, network request redirection, and content management. Content management and global network traffic management are The core of CDN lies. In this disclosure, the CDN network uses the proximity of the second client device 105 and the load of each content distribution network server 103 to determine data transmission and distribution. The CDN network ensures that audio and video data is transmitted to the second in an extremely efficient manner. At 105 user-end equipment.

It is worth mentioning that in the prior art, when the CDN network performs data forwarding, the audio and video data will be further processed so as to be applicable to different user terminals. In this disclosure, for simplicity, the second audio The audio and video data generated by the video data packet processed by the CDN network is still called the second audio and video data packet.

In the live broadcast application scenario, ordinary users can use the second client device 105 to interact with the content distribution network server 103 through the network 104 to receive or send audio and video data or other various data. Ordinary users can also use the second user-end device 105 to directly interact with the first user-end device 101 through the network 104 to receive or send signaling information.

The second user end device 105 may obtain at least one second audio/video data packet and at least one signaling information, for example; the second user end device 105 may obtain the first time stamp data, for example, by parsing the signaling information; the second user end The device 105 may, for example, obtain the second time stamp data by parsing the second audio and video data packet; the second client device 105 may, for example, convert the at least one time stamp data based on the first time stamp data and the second time stamp data. The signaling information and the at least one second audio and video data packet are time-sorted so as to be played sequentially.

It should be noted that the method for synchronous playback of audio and video data provided by the embodiments of the present disclosure can be executed by the first client device 101, the transfer server 102, and the second client device 105. Accordingly, the device for synchronous playback of audio and video data can be set In the first client device 101, the transfer server 102, and the second client device 105. More specifically, a device provided for the host user to generate audio and video data is located in the first client device 101, and a device provided for ordinary users to accept and play the video and audio data is located in the second client device 105.

Fig. 2 is a schematic diagram showing an application scenario of a method and device for synchronously playing audio and video data according to an exemplary embodiment. As shown in Figure 2, in the live broadcast application scenario, the host user generates audio and video data through the first user terminal, and then sends the audio and video data to the network. Ordinary users receive the audio and video data through the second user terminal, and then send the audio and video data to the network. The video data is played in the second client.

More specifically, in the live broadcast scene of online teaching on the Internet, the student uses the second client (the second client can be a computer or mobile phone or other devices that can play audio and video) and the teacher uses the first client (The first user terminal can be a computer or a mobile phone or other device with a camera), students and teachers can use their respective terminals to use the Internet to realize the process of remote video teaching. Among them, for any user terminal, the received image and sound of the other party may be the audio and video data in the disclosure, and the red envelope received may be the signaling information in the disclosure.

According to the audio and video data synchronization playback system of the present disclosure, a second audio and video data packet is generated from the first audio and video data packet and time stamp data; and the second audio and video data packet is transmitted to the second audio and video data packet through a content distribution network The user terminal can solve the problem that the audio and video data is not synchronized with the signaling information sent by the user in the video live broadcast process by displaying.

Fig. 3 is a flowchart showing a method for synchronously playing audio and video data according to an exemplary embodiment. The method 30 for synchronously playing audio and video data can be applied to the first client. The method 30 for synchronously playing audio and video data at least includes steps S302 to S304.

As shown in FIG. 3, in S302, audio and video data is acquired, where the audio and video data includes at least one first audio and video data packet and at least one signaling information. Among them, the first video data may be real-time video data generated by the host user, and the signaling information may be information such as red packets generated by the host user.

In S304, the at least one first audio and video data packet is sent to the at least one second user terminal through the relay server based on the web instant messaging protocol, and the at least one signaling information is sent to the at least one second user terminal based on the transmission control protocol. A second user terminal for playing audio and video data.

This may include, for example, sending the first audio and video data packet to the second client through the relay server based on the Web Real-Time Communication (WebRTC) protocol to display live video data.

Among them, the web instant messaging technology is an API (Application Programming Interface) that supports web browsers for real-time voice or video conversations. WebRTC implements web-based video conferences with the purpose of providing simple javascript through the browser You can achieve real-time communication capabilities. WebRTC provides the core technology of video conferencing, including audio and video collection, encoding and decoding, network transmission, display and other functions, and also supports cross-platform, including linux, windows, mac, android, etc.

This may include, for example, sending the signaling information to the second user terminal based on the Transmission Control Protocol (TCP) to display live video data.

Among them, the TCP protocol is a connection-oriented, reliable, byte stream-based transport layer communication protocol, which is defined by IETF RFC793. User Datagram Protocol (UDP) is another important transmission protocol in the same layer. In the Internet protocol suite (Internet protocol suite), the TCP layer is an intermediate layer above the IP layer and below the application layer. The application layers of different hosts often need reliable, pipe-like connections, but the IP layer does not provide such a flow mechanism, but provides unreliable packet exchange.

Fig. 4 is a flowchart showing a method for synchronously playing audio and video data according to an exemplary embodiment. The method 40 for synchronously playing audio and video data can be applied to the transfer server. The method 40 for synchronously playing audio and video data at least includes steps S402 to S406.

As shown in FIG. 4, in S402, the first audio and video data packet from the first user terminal is acquired.

In S404, a second audio and video data packet is generated from the first audio and video data packet and the time stamp data. The method includes: converting the first audio and video data packet to generate a target audio and video data packet; and generating the second audio and video data packet through the target audio and video data packet and the time stamp data.

In one embodiment, generating the second audio and video data packet from the target audio and video data packet and the time stamp data includes: generating the time stamp data according to the universal standard time; generating supplements from the time stamp data Enhancement information; and generating the second audio and video data packet through the target audio and video data packet and the supplementary enhancement information. The supplementary enhancement information may be SEI enhancement information, and the SEI carries the UTC (Universal Standard Time) timestamp when the audio and video data packet is received, and transmits the audio and video data to the CDN network.

In one embodiment, generating the second audio and video data packet from the target audio and video data packet and the time stamp data includes: generating the time stamp data according to Universal Standard Time; and writing the time stamp data Into the file name of the target audio and video data packet to generate the second audio and video data packet. In order to avoid the situation that some servers do not support SEI enhanced information, the timestamp can also be written into the file name of the audio and video data packet in digital form, and the present disclosure is not limited to this.

In an embodiment, converting the first audio and video data packet to generate a target audio and video data packet includes: converting the first audio and video data packet based on real-time communication technology to generate a real-time message transmission protocol based Target audio and video data packets.

In an embodiment, converting the first audio and video data packet to generate a target audio and video data packet includes: performing format conversion processing on the data format of the first audio and video data packet to generate the target audio and video data. Data packet; and/or protocol conversion processing of the communication protocol of the first audio and video data packet to generate the target audio and video data packet; and/or protocol conversion of the transmission protocol of the first audio and video data packet Conversion processing to generate the target audio and video data packet.

In one embodiment, performing format conversion processing on the data format of the first audio and video data packet includes: performing format conversion processing on the video format in the first audio and video data packet; and/or performing format conversion processing on the first audio and video data packet; The audio format in the audio and video data packet undergoes format conversion processing.

Specifically, for example, the communication protocol of the first audio and video data packet can be converted from the web instant messaging protocol (WebRTC) to the real-time message protocol (RTMP); the transmission protocol of the first audio and video data packet can be transferred to the user Datagram protocol (UDP, User Datagram Protocol) is converted to Transmission Control Protocol (Transmission Control Protocol, TCP); the video format in the first audio and video data packet is converted from VP8 to H264; the first audio and video data is converted The audio format in the package is converted from opus to AAC.

In S406, the second audio and video data packet is transmitted to the second user terminal through the content distribution network for display.

Fig. 5 is a flowchart showing a method for synchronously playing audio and video data according to an exemplary embodiment. The method 50 for synchronously playing audio and video data can be applied to a second client. The method 50 for synchronously playing audio and video data at least includes steps S502 to S508.

As shown in FIG. 5, in S502, at least one second audio and video data packet and at least one signaling information are acquired.

In S504, the first time stamp data is obtained by parsing the signaling information.

In S506, the second time stamp data is obtained by parsing the second audio and video data packet.

In S508, the at least one signaling information and the at least one second audio and video data packet are time-sorted based on the first time stamp data and the second time stamp data, so as to be played sequentially.

In an embodiment, for example, a playback time axis can be generated based on the second time stamp data; a playback queue of the second audio and video data packet can be generated based on the playback time axis; and based on the first time stamp data Insert the signaling information into the play queue for display.

In one embodiment, inserting the signaling information into the play queue for display based on the first time stamp data includes: generating a display time from the first time stamp data; and comparing the display time with The playing time in the playing time axis is compared, and the signaling information is inserted into the playing queue for display according to the comparison result.

Fig. 6 is a flowchart showing a system for displaying live video data according to another exemplary embodiment. The method 60 for synchronous playback of audio and video data corresponding to FIG. 6 describes the entire working process of the video live data display system.

As shown in FIG. 6, in S602, the first user end sends signaling information to the second user end.

In S604, the first user terminal sends the first audio and video data packet to the transit server.

In S606, the transit server processes the first audio and video data packet to generate a second audio and video data packet.

In S608, the transit server sends the second audio and video data packet to the CDN.

In S610, the CDN processes the second audio data.

In S612, the CDN sends the second audio and video data packet to the second client.

In S614, the second user segment analyzes the signaling information and the second audio and video data packet, and plays according to the analysis result.

According to the method for synchronous playback of audio and video data of the present disclosure, a second audio and video data packet is generated from the first audio and video data packet and time stamp data; and the second audio and video data packet is transmitted to the second audio and video data packet through a content distribution network. The user terminal can solve the problem that the audio and video data is not synchronized with the signaling information sent by the user in the video live broadcast process by displaying.

It should be clearly understood that the present disclosure describes how to form and use specific examples, but the principles of the present disclosure are not limited to any details of these examples. On the contrary, based on the teachings of the contents disclosed in the present disclosure, these principles can be applied to many other embodiments.

Those skilled in the art can understand that all or part of the steps for implementing the above-mentioned embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, it executes the above-mentioned functions defined by the above-mentioned method provided in the present disclosure. The program can be stored in a computer-readable storage medium, which can be a read-only memory, a magnetic disk, or an optical disk.

In addition, it should be noted that the above-mentioned drawings are only schematic illustrations of the processing included in the method according to the exemplary embodiments of the present disclosure, and are not intended for limitation. It is easy to understand that the processing shown in the above drawings does not indicate or limit the time sequence of these processings. In addition, it is easy to understand that these processes can be executed synchronously or asynchronously in multiple modules, for example.

The following are device embodiments of the present disclosure, which can be used to implement the method embodiments of the present disclosure. For details that are not disclosed in the device embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

Fig. 7 is a block diagram showing a device for synchronously playing audio and video data according to an exemplary embodiment. As shown in FIG. 7, the device 70 for synchronous playback of audio and video data includes: a data acquisition module 702 and a data transmission module 704.

The data acquisition module 702 is configured to acquire audio and video data. The audio and video data includes at least one first audio and video data packet and at least one signaling information; wherein the first video data can be real-time video data generated by the host user, and the signaling The information can be information such as red envelopes generated by the host user.

The data sending module 704 is configured to send the at least one first audio and video data packet to at least one second user terminal through the transit server based on the web instant messaging protocol, and send the at least one signaling information to the at least one signaling information based on the transmission control protocol. Said at least one second user terminal for playing audio and video data.

Fig. 8 is a block diagram showing a device for synchronously playing audio and video data according to another exemplary embodiment. As shown in FIG. 8, the device 80 for synchronously playing audio and video data includes: a data transfer module 802, an information addition module 804, and a data transmission module 806.

The data transfer module 802 is configured to obtain the first audio and video data packet from the first user terminal;

The information adding module 804 is configured to generate a second audio and video data packet from the first audio and video data packet and the time stamp data;

For example, the first audio and video data packet is converted to generate a target audio and video data packet; and the second audio and video data packet is generated from the target audio and video data packet and the time stamp data.

For example, the time stamp data may be generated according to the universal standard time; and the time stamp data may be written into the file name of the target audio and video data packet to generate the second audio and video data packet. In order to avoid the situation that some servers do not support SEI enhanced information, the timestamp may also be written into the file name of the audio and video data packet in digital form, and the present disclosure is not limited to this.

The third transmission module 806 is configured to transmit the second audio and video data packet to the second user terminal through the content distribution network for playback.

Fig. 9 is a block diagram showing a device for synchronously playing audio and video data according to another exemplary embodiment. As shown in FIG. 9, the device 90 for synchronously playing audio and video data includes: a data receiving module 902, a first time module 904, a second time module 906, and a time sorting module 908.

The data receiving module 902 is configured to obtain at least one second audio and video data packet and at least one signaling information;

The first time module 904 is configured to obtain first time stamp data by parsing the signaling information;

The second time module 906 is configured to obtain second time stamp data by parsing the second audio and video data packet; and

The time sorting module 908 is configured to sort the at least one signaling information and the at least one second audio and video data packet in time based on the first time stamp data and the second time stamp data, so as to be played sequentially.

In one embodiment, for example, a playback time axis can be generated based on the second time stamp data; a playback queue of the second audio and video data packet can be generated based on the playback time axis; and based on the first time stamp data Insert the signaling information into the play queue for display.

According to the audio and video data synchronization playback device of the present disclosure, a second audio and video data packet is generated from the first audio and video data packet and time stamp data; and the second audio and video data packet is transmitted to the second audio and video data packet through a content distribution network. The user terminal can solve the problem that the audio and video data is not synchronized with the signaling information sent by the user in the video live broadcast process by displaying.

The electronic device 200 according to this embodiment of the present disclosure will be described below with reference to FIG. 10. The electronic device 200 shown in FIG. 10 is only an example, and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

As shown in FIG. 10, the electronic device 200 is represented in the form of a general-purpose computing device. The components of the electronic device 200 may include but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, and the like.

Wherein, the storage unit stores program code, and the program code can be executed by the processing unit 210, so that the processing unit 210 executes the various exemplary methods described in the above-mentioned electronic prescription circulation processing method section of this specification. Implementation steps. For example, the processing unit 210 may execute the steps shown in FIG. 3, FIG. 4, FIG. 5, and FIG. 6.

The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 2201 and/or a cache storage unit 2202, and may further include a read-only storage unit (ROM) 2203.

The storage unit 220 may also include a program/utility tool 2204 having a set of (at least one) program module 2205. Such program module 2205 includes but is not limited to: an operating system, one or more application programs, other program modules, and programs. Data, each of these examples or some combination may include the realization of a network environment.

The bus 230 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local area using any bus structure among multiple bus structures. bus.

The electronic device 200 can also communicate with one or more external devices 300 (such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 200, and/or communicate with Any device (such as a router, modem, etc.) that enables the electronic device 200 to communicate with one or more other computing devices. This communication can be performed through an input/output (I/O) interface 250. In addition, the electronic device 200 may also communicate with one or more networks (for example, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through the network adapter 260. The network adapter 260 can communicate with other modules of the electronic device 200 through the bus 230. It should be understood that although not shown in the figure, other hardware and/or software modules can be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.

Through the description of the foregoing embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, as shown in FIG. 11, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, or mobile hard disk). Etc.) or on the network, including several instructions to make a computing device (which may be a personal computer, a server, or a network device, etc.) execute the above method according to the embodiments of the present disclosure.

The software product can use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Type programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

The computer-readable storage medium may include a data signal propagated in baseband or as a part of a carrier wave, and readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than the readable storage medium, and the readable medium can send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

The program code used to perform the operations of the present disclosure can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural styles. Programming language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computing device, partly on the user's device, executed as an independent software package, partly on the user's computing device and partly executed on the remote computing device, or entirely on the remote computing device or server Executed on. In the case of a remote computing device, the remote computing device can be connected to a user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using Internet service providers) Business to connect via the Internet).

The aforementioned computer-readable medium carries one or more programs, and when the aforementioned one or more programs are executed by a device, the computer-readable medium realizes the following functions: acquiring the first audio and video data packet from the first user terminal; Generate a second audio and video data packet through the first audio and video data packet and the time stamp data; and transmit the second audio and video data packet to a second user terminal through a content distribution network for playback.

Those skilled in the art can understand that the foregoing modules can be distributed in the device according to the description of the embodiment, or corresponding changes can be made in one or more devices that are uniquely different from this embodiment. The modules of the above-mentioned embodiments can be combined into one module or further divided into multiple sub-modules.

Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described here can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , Including several instructions to make a computing device (which may be a personal computer, server, mobile terminal, or network device, etc.) execute the method according to the embodiments of the present disclosure.

The exemplary embodiments of the present disclosure have been specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present disclosure intends to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

A method for synchronously playing audio and video data, characterized in that it comprises:

Acquiring the first audio and video data packet from the first user terminal;

Generating a second audio and video data packet from the first audio and video data packet and time stamp data; and

The second audio and video data packet is transmitted to the second user terminal through the content distribution network for playback.
The method according to claim 1, wherein generating a second audio and video data packet from the first audio and video data packet and time stamp data comprises:

Performing conversion processing on the first audio and video data packet to generate a target audio and video data packet; and

The second audio and video data packet is generated through the target audio and video data packet and the time stamp data.
3. The method of claim 2, wherein generating the second audio and video data packet by using the target audio and video data packet and the time stamp data comprises:

Generating the timestamp data through universal standard time;

Generating supplementary enhanced information from the timestamp data; and

The second audio and video data packet is generated by using the target audio and video data packet and the supplementary enhancement information.
3. The method of claim 2, wherein generating the second audio and video data packet by using the target audio and video data packet and the time stamp data comprises:

The time stamp data is generated by UTC; and

The time stamp data is written into the file name of the target audio and video data packet to generate the second audio and video data packet.
The method according to claim 2, wherein converting the first audio and video data packet to generate a target audio and video data packet comprises:

Converting the first audio and video data packet based on the real-time communication technology to generate a target audio and video data packet based on the real-time message transmission protocol.
The method according to claim 2, wherein converting the first audio and video data packet to generate a target audio and video data packet comprises:

Performing format conversion processing on the data format of the first audio and video data packet to generate the target audio and video data packet; and/or

Perform protocol conversion processing on the communication protocol of the first audio and video data packet to generate the target audio and video data packet; and/or

Performing protocol conversion processing on the transmission protocol of the first audio and video data packet to generate the target audio and video data packet.
7. The method according to claim 6, wherein performing format conversion processing on the data format of the first audio and video data packet comprises:

Perform format conversion processing on the video format in the first audio and video data packet; and/or

Perform format conversion processing on the audio format in the first audio and video data packet.
A method for synchronously playing audio and video data, characterized in that it comprises:

Acquiring at least one second audio and video data packet and at least one signaling information;

Obtain the first time stamp data by analyzing the signaling information;

Obtaining second time stamp data by parsing the second audio and video data packet; and

Based on the first time stamp data and the second time stamp data, the at least one signaling information and the at least one second audio and video data packet are time-sorted for sequential playback.
The method according to claim 8, wherein the at least one signaling information and the at least one second audio/video data packet are combined based on the first time stamp data and the second time stamp data Time sequence, including:

Generating a play queue of the at least one second audio and video data packet through the play time axis determined by the second time stamp data; and

Inserting the at least one signaling information into the play queue based on the first time stamp data.
The method of claim 9, wherein the inserting the at least one signaling information into the play queue based on the first time stamp data comprises:

Generating display time from the first time stamp data;

Compare the display time with the play time in the play timeline, and

Insert the at least one signaling information into the play queue according to the comparison result.
A method for synchronously playing audio and video data, characterized in that it comprises:

Acquiring audio and video data, where the audio and video data includes at least one first audio and video data packet and at least one signaling information; and

The at least one first audio and video data packet is sent to at least one second user terminal through the relay server based on the webpage instant messaging protocol, and the at least one signaling information is sent to the at least one second user based on the transmission control protocol Terminal for audio and video data playback.
A synchronous playback device for audio and video data, characterized in that it comprises:

The data transfer module is used to obtain the first audio and video data packet from the first user terminal;

An information adding module, configured to generate a second audio and video data packet from the first audio and video data packet and time stamp data; and

The data transmission module is configured to transmit the second audio and video data packet to the second user terminal through the content distribution network for playback.
A synchronous playback device for audio and video data, characterized in that it comprises:

A data receiving module, configured to obtain at least one second audio and video data packet and at least one signaling information;

The first time module is configured to obtain the first time stamp data by parsing the signaling information;

The second time module is configured to obtain second time stamp data by parsing the second audio and video data packet; and

The time ordering module is configured to time order the at least one signaling information and the at least one second audio and video data packet based on the first time stamp data and the second time stamp data, so as to be played sequentially.
A synchronous playback device for audio and video data, characterized in that it comprises:

A data acquisition module for acquiring audio and video data, the audio and video data including at least one first audio and video data packet and at least one signaling information; and

The data sending module is configured to send the at least one first audio and video data packet to the at least one second user terminal through the relay server based on the web instant messaging protocol, and send the at least one signaling information to the at least one signaling information based on the transmission control protocol Said at least one second user terminal for playing audio and video data.
A synchronous playback system for audio and video data, characterized in that it comprises:

The first user terminal is configured to acquire audio and video data, where the audio and video data includes at least one first audio and video data packet and at least one signaling information; and the at least one first audio and video data packet is transmitted based on the web instant messaging protocol Sending to at least one second user terminal through a transit server, and sending the at least one signaling information to the at least one second user terminal based on a transmission control protocol, so as to play audio and video data;

The relay server is configured to obtain the first audio and video data packet from the first user terminal; generate a second audio and video data packet from the first audio and video data packet and the time stamp data; and pass the second audio and video data packet through The content distribution network is transmitted to the second client for playback; and

A content distribution network server, configured to transmit the second video data packet to the second client; and

The second user terminal is configured to obtain at least one second audio and video data packet and at least one signaling information; obtain the first time stamp data by analyzing the signaling information; obtain the second time stamp data by analyzing the second audio and video data packet. Time stamp data; and based on the first time stamp data and the second time stamp data, the at least one signaling information and the at least one second audio and video data packet are time-sorted, so as to be played sequentially.
An electronic device, characterized in that it comprises:

One or more processors;

Storage device for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-3 or 4-10 or 11-13 .
A computer-readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method according to any one of claims 1-3 or 4-10 or 11-13.