CN110213652B - Audio and video data transmission method and device and storage medium - Google Patents

Audio and video data transmission method and device and storage medium Download PDF

Info

Publication number
CN110213652B
CN110213652B CN201810333639.XA CN201810333639A CN110213652B CN 110213652 B CN110213652 B CN 110213652B CN 201810333639 A CN201810333639 A CN 201810333639A CN 110213652 B CN110213652 B CN 110213652B
Authority
CN
China
Prior art keywords
audio
video data
client
transcoded
video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810333639.XA
Other languages
Chinese (zh)
Other versions
CN110213652A (en
Inventor
黄柳文
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201810333639.XA priority Critical patent/CN110213652B/en
Publication of CN110213652A publication Critical patent/CN110213652A/en
Application granted granted Critical
Publication of CN110213652B publication Critical patent/CN110213652B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/40—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Telephonic Communication Services (AREA)

Abstract

The embodiment of the invention discloses an audio and video data transmission method, an audio and video data transmission device and a storage medium, wherein the audio and video data transmitted by a transmitting client can be received in audio and video sessions among a plurality of clients of different types, and the transmitting client is any one of the clients; acquiring coding formats required by all receiving clients, wherein the receiving clients are clients except for a sending client in a plurality of clients; transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; and respectively transmitting the transcoded audio and video data corresponding to each receiving client. The scheme realizes the audio and video data interaction of different types of clients, overcomes the limitation of the existing audio and video data transmission, and greatly improves the flexibility of the audio and video data transmission.

Description

Audio and video data transmission method and device and storage medium
Technical Field
The invention relates to the technical field of internet, in particular to an audio and video data transmission method, an audio and video data transmission device and a storage medium.
Background
With the development of internet technology, users not only meet the communication and social contact of characters, but also have stronger and stronger requirements for real-time voice and video, so that the users are more and more popularized in information interaction through audio and video sessions, and the users can join different audio and video sessions through terminals according to own requirements so as to perform information interaction and the like in the audio and video sessions.
In the prior art, multiple terminals need to join in audio and video sessions through the same client, so that in the same audio and video session, each terminal can perform audio and video data interaction through the same client. For example, the terminal a, the terminal B, and the terminal C may download and install a client in which a Software Development Kit (SDK) is built in advance, and then join an audio/video session based on the client; therefore, the audio and video data can be sent in the audio and video session or received, and the like, so as to realize information interaction.
In the research and practice processes of the prior art, the inventor of the present invention finds that, when the existing terminal performs audio/video data interaction, the same client is required to join the same audio/video session, so that the limitation of performing audio/video data interaction is large and the universality is not high.
Disclosure of Invention
The embodiment of the invention provides an audio and video data transmission method, an audio and video data transmission device and a storage medium, and aims to improve the flexibility of audio and video data transmission.
In order to solve the above technical problems, embodiments of the present invention provide the following technical solutions:
an audio and video data transmission method comprises the following steps:
in audio and video sessions among a plurality of clients of different types, receiving audio and video data sent by a sending client, wherein the sending client is any one of the clients;
acquiring a coding format required by each receiving client, wherein the receiving client is a client except the sending client in the plurality of clients;
transcoding the audio and video data respectively according to the coding formats required by the receiving clients to obtain transcoded audio and video data corresponding to the receiving clients;
and respectively transmitting the transcoded audio and video data corresponding to each receiving client.
An audio-video data transmission apparatus comprising:
the first receiving unit is used for receiving audio and video data sent by a sending client in audio and video sessions among a plurality of clients of different types, wherein the sending client is any one of the clients;
an obtaining unit, configured to obtain an encoding format required by each receiving client, where the receiving client is a client of the multiple clients except the sending client;
the transcoding unit is used for transcoding the audio and video data respectively according to the coding formats required by the receiving clients to obtain transcoded audio and video data corresponding to the receiving clients;
and the transmission unit is used for respectively transmitting the transcoded audio and video data corresponding to each receiving client.
A storage medium, wherein the storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the audio and video data transmission method.
The embodiment of the invention can receive and send the audio and video data sent by the client in the audio and video session among a plurality of clients of different types, wherein the sending client is any one of the clients; acquiring a coding format required by each receiving client, wherein the receiving client is a client except for the sending client in the plurality of clients; then transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; and respectively transmitting the transcoded audio and video data corresponding to each receiving client. According to the scheme, multiple clients of different types can join the same audio and video session, audio and video data are transcoded in the audio and video session according to the coding formats required by the receiving clients, and the transcoded audio and video data are transmitted to the receiving clients respectively, so that the audio and video data interaction of the clients of different types in the same audio and video session is realized, the limitation that the same client can join the same audio and video session to transmit the audio and video data in the prior art is overcome, and the flexibility of audio and video data transmission is greatly improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of an implementation environment related to audio-video data transmission provided by an embodiment of the present invention;
fig. 2 is a schematic diagram of another implementation environment related to audio-video data transmission provided by the embodiment of the invention;
fig. 3 is a scene schematic diagram of an audio/video data transmission system provided in an embodiment of the present invention;
fig. 4 is a schematic flow chart of an audio/video data transmission method provided by an embodiment of the present invention;
fig. 5 is another schematic flow chart of an audio-video data transmission method provided by an embodiment of the present invention;
fig. 6 is an architecture diagram of a relay server for audio and video data transmission according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an audio/video data transmission device according to an embodiment of the present invention;
fig. 8 is another schematic structural diagram of an audio-video data transmission device provided in an embodiment of the present invention;
fig. 9 is another schematic structural diagram of an audio-video data transmission device provided in an embodiment of the present invention;
fig. 10 is another schematic structural diagram of an audio-video data transmission device provided in an embodiment of the present invention;
fig. 11 is a schematic structural diagram of a server according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides an audio and video data transmission method, an audio and video data transmission device and a storage medium.
Referring to fig. 1, fig. 1 is a schematic view of an implementation environment related to an audio/video data transmission method provided by an embodiment of the present invention, where the implementation environment may include: a sending client 101, a receiving client 102, a login server 103, a signaling server 104, a query server 105, a relay server 106, a policy server 107, an interface machine 108, and the like.
The sending client 101 may be an SDK client, an applet client, a traditional Telephone Network (PSTN) client, a Web client (also referred to as an H5 client, for example, a Public client and a browser client), and the like, where the sending client 101 may be integrated on a terminal, and the terminal where the sending client 101 is located may be a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like, and the embodiment of the present invention does not specifically limit the product type of the sending client 101. A module supporting a Web Real-Time Communication (WebRTC) function may be built in the terminal where the sending client 101 is located, so that the terminal may establish a network connection with the relay server 106, the interface unit 108, and the like based on the WebRTC function module, and the receiving client 102 may join in an audio/video session. Or, the sending client 101 may be a client installed with a built-in SDK, or the sending client 101 may be installed with an SDK component, so that an audio and video session may be added; and so on.
The receiving client 102 may be an SDK client, an applet client, a PSTN client, or a Web client, and the Web client may include a public account client, a browser client, and the like, and the receiving client 102 may be integrated on a terminal, and the terminal where the receiving client 102 is located may be a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like. The receiving client 102 may be a client installed with a built-in SDK, or the receiving client 102 may be installed with an SDK component so that an audio-video session may be joined. Or, a module supporting the WebRTC function may be built in the terminal where the receiving client 102 is located, so that the terminal may establish a network connection with the transfer server 106, the interface unit 108, and the like based on the module supporting the WebRTC function, and the receiving client 102 may join in the audio/video session.
The login server 103 can be a set of universal account login components provided by a developer for quickly completing account integration access to audio/video or instant messaging cloud services.
The signaling server 104 may have built-in modules that support WebRTC functionality, and the functionality provided by the signaling server 104 may include:
(1) providing access and login functions, so that clients such as an applet client, a PSTN client, a Web client and the like can access and login audio and video sessions; (2) providing a signaling transmission function, so that signaling can be transmitted between clients such as an applet client, a PSTN client, and a Web client, and the transfer server 106; (3) a transfer server 106 that provides an allocation function so as to allocate network connections to clients such as an applet client, a PSTN client, and a Web client; (4) and a data negotiation issuing function is provided, so that the negotiation issuing of audio and video initial parameters, the negotiation of a connection channel, the negotiation issuing of security parameters and the like can be performed between clients such as an applet client, a PSTN client, a Web client and the like and the transfer server 106.
Query server 105 is primarily responsible for providing a unified query service that includes queries for user profiles, relationship chains, notes, groupings, group relationship chains, user-generated information, and the like.
The transit server 106 has built-in modules supporting WebRTC functionality, and the functions provided by the transit server 106 may include:
(1) simulating WebRTC by facing clients such as an applet client, a PSTN client, a Web client and the like, and establishing network connection with the clients such as the applet client, the PSTN client, the Web client and the like, wherein the network connection process can comprise Interactive Connectivity Establishment (ICE), key negotiation of a data packet Transport Layer Security Protocol (DTLS), transfer transmission of audio and video data of a Secure Real-time Transport Protocol (SRTP) and Control of audio and video quality of a Real-time Transport Control Protocol (RTCP); (2) the interface-oriented machine simulates an SDK client and establishes connection with an audio and video session system, including adding audio and video sessions, transmitting audio and video data and the like; (3) linking signaling for adding the client side such as the applet client side, the PSTN client side, the Web client side and the like into the audio and video session, signaling for quitting the audio and video session, state notification and the like; (4) and transcoding, transcoding and packaging the audio and video data of the applet client, the PSTN client, the Web client and the SDK client, and the like.
Policy server 107, the functions provided by policy server 107 may include: (1) the system is responsible for authentication, signature encryption, security confirmation and the like of session requests; (2) according to the position information of the client, the position information and the load information of the signaling server 104, the transit server 106, the interface machine 108 and the like, the interface machine 108, the streaming controller and the like are distributed to the client nearby, so that the same audio and video session and the same operator can be regarded as the client to be gathered to the same interface machine 108 as much as possible.
The interface engine 108 may be configured to forward audio and video data and signaling in the audio and video session system.
The sending client 101 may be connected to the login server 103, the signaling server 104, and the relay server 106 via a wired network or a wireless network, the login server 103 may be connected to the signaling server 104 via a wired network or a wireless network, the signaling server 104 may be connected to the relay server 106 via a wired network or a wireless network, the query server 105 may be connected to the relay server 106 via a wired network or a wireless network, and the relay server 106 may be connected to the policy server 107, the interface unit 108, and the receiving client 102 via a wired network or a wireless network.
Referring to fig. 2, fig. 2 is a schematic diagram of another implementation environment related to the audio/video data transmission method provided by the embodiment of the present invention, where the implementation environment may include: a sending client 101, a receiving client 102, a server 109, and the like.
The sending client 101 has the same product type and function as the sending client 101 shown in fig. 1, and is not described herein again. The receiving client 102 has the same product type and function as the receiving client 102 shown in fig. 1, and is not described herein again. The server 109 has all functions of the login server 103, the signaling server 104, the query server 105, the relay server 106, the policy server 107, and the interface unit 108 in fig. 1, and for details, reference is made to the description of the login server 103, the signaling server 104, the query server 105, the relay server 106, the policy server 107, and the interface unit 108 in fig. 1, which is not described herein again.
It should be noted that the schematic diagrams of the implementation environments related to the audio-video data transmission methods shown in fig. 1 and fig. 2 are only an example, and the implementation environment of audio-video data transmission described in the embodiment of the present invention is to more clearly illustrate the technical solution of the embodiment of the present invention, and does not form a limitation on the technical solution provided in the embodiment of the present invention.
Referring to fig. 3, based on the implementation environment shown in fig. 1, fig. 3 is a schematic view of a scene of an audio and video data transmission system provided in an embodiment of the present invention, where a Web client may include a public client, a browser client, and the like, and an audio and video session may be established between different types of clients, such as an applet client, a public client, a browser client, a PSTN client, and an SDK client, and in the audio and video session, any one of the applet client, the public client, the browser client, the PSTN client, and the SDK client may be a sending client or a receiving client.
For the SDK client, when an application needs to be added to the audio/video session due to a service requirement, the SDK client may send a session request to the policy server, where the session request may carry an audio/video session identifier, a login account, an Internet Protocol (IP) address interconnected between networks of the SDK client, and the like. When receiving the session request, the policy server may query, according to the IP address of the SDK client, a data center to which the IP address of the SDK client belongs from an IP library, where the IP library may store a correspondence between the IP address and the data center, and the data center is a data forwarding and processing center divided for reducing data processing pressure of the server. The policy server can also inquire information such as user data, a relationship chain, a group relationship chain, position information and the like of the SDK client from the inquiry server according to the login account; the strategy scheduling server can also carry out login verification on the SDK client through the login server according to the login account. When the verification of the SDK client passes, the policy server can judge whether the audio/video session corresponding to the audio/video session identifier exists according to the audio/video session identifier.
If the audio and video session corresponding to the audio and video session identifier exists, the policy server can determine a flow control IP corresponding to the audio and video session. At this time, the policy scheduling server may allocate an interface machine and the like to the SDK client according to the queried information of the SDK client and the data center to which the SDK client belongs. If the audio/video session corresponding to the audio/video session identifier does not exist, the policy scheduling server can allocate a flow control machine, an interface machine and the like to the SDK client according to the inquired information of the SDK client, the data center to which the SDK client belongs and the like. And the strategy scheduling server sends the distributed IP of the flow control machine and the IP of the interface machine to the SDK client, and the SDK client can add the received IP of the flow control machine and the interface machine into the audio and video conversation and establish connection with the flow control machine and the interface machine.
When the audio and video data need to be sent to other clients in the audio and video session, the SDK client can send the audio and video data to the interface machine, and then send the audio and video data to other clients in the audio and video session through the interface machine; when receiving audio and video data sent by other clients in the audio and video session, the interface machine connected with the SDK client can send the received audio and video data to the SDK client.
For the Web client, when the Web client needs to apply for joining the audio/video session due to the service requirement, the Web client can send a login request to the login server, the login server performs login verification on the Web client, and when the verification is passed, the Web client can complete login.
After the login is completed, the Web client may send a data transmission capability negotiation request to the signaling server, where the data transmission capability negotiation request may carry an audio data format and a video data format supported by the Web client, a Session Description Protocol (SDP), a channel (e.g., an IP and a port for receiving and transmitting audio and video data), and the like, and the signaling server sends the data transmission capability negotiation request to the relay server. When receiving the data transmission capability negotiation request, the transit server may store an audio data format, a video data format, and the like supported by the Web client, determine a data format when audio and video data transmission is performed with the Web client according to the audio data format and the video data format supported by the Web client and the audio data format and the video data format supported by the transit server, and send a data transmission capability negotiation response to the signaling server, where the data transmission capability negotiation response includes the data format when audio and video data transmission is performed with the Web client, and the like. When receiving the data transmission capability negotiation response, the signaling server may transmit the data transmission capability negotiation response to the Web client.
The Web client may also send a session request to the signaling server, where the session request may carry an audio/video session identifier, a login account, an IP address of the Web client, and the like. The signaling server forwards the session request to the transfer server, and after receiving the session request, the transfer server can replace the account number for the Web client so as to unify the account number types of the clients in the audio and video session. After changing the account, the transit server may forward the session request to the policy scheduling server. When receiving the session request, the policy server may query the data center to which the IP address of the Web client belongs from the IP library according to the IP address of the Web client, and may also query information such as user information, a relationship chain, a group relationship chain, and location information of the Web client from the query server according to the login account.
The policy server can judge whether an audio/video session corresponding to the audio/video session identifier exists according to the audio/video session identifier, and if the audio/video session corresponding to the audio/video session identifier exists, the policy server can determine a flow control IP corresponding to the audio/video session. In this case, the policy scheduling server may allocate an interface machine and the like to the Web client according to the queried information of the Web client and the data center to which the Web client belongs. If the audio/video session corresponding to the audio/video session identifier does not exist, the policy scheduling server can allocate a flow control machine, an interface machine and the like to the Web client according to the inquired information of the Web client, the data center to which the Web client belongs and the like.
The strategy scheduling server can send the distributed IP of the flow control machine and the IP of the interface machine to the transfer server. After receiving the stream control machine IP and the interface machine IP, the relay server may establish connection with the stream control machine corresponding to the stream control machine IP and with the interface machine corresponding to the interface machine IP, at this time, the relay server may allocate a relay IP for the Web client and send the relay IP to the Web client, and the Web client may establish secure connection with the relay server according to the relay IP. Therefore, the network connection between the signaling server and the transfer server can be used for joining the audio and video session based on the network connection established between the Web client and the transfer server and the network connection established between the transfer server and the interface machine.
When the audio and video data need to be sent to other clients in the audio and video session, the Web client can send the audio and video data to the transfer server, the transfer server sends the audio and video data to the interface machine, and then the audio and video data are sent to other clients in the audio and video session through the interface machine. When receiving audio and video data sent by other clients in the audio and video session, the interface machine connected with the Web client can send the received audio and video data to the transfer server, and then the transfer server sends the audio and video data to the Web client.
It should be noted that, for the applet client, the PSTN client and other clients, when the application needs to be added to the audio/video session due to the service requirement, the processes of adding the audio/video session and transmitting or receiving the audio/video data are added, and the processes of adding the audio/video session and transmitting or receiving the audio/video data by the Web client are similar, which is not described herein again.
In the process of transmitting audio and video data, the audio and video data transmission system can also manage audio and video sessions through the flow control machine and adjust various audio and video parameters and audio and video routing in the process of transmitting the audio and video data, so that the adjustment of the transmission quality of the audio and video data is realized. The specific adjusting process can be as follows: the flow control machine acquires information such as various audio and video parameters and audio and video routing from the policy server, and then sends an adjusting instruction to the interface machine in a signaling form, and the interface machine adjusts audio and video data sent by clients such as the SDK client, the applet client, the PSTN client, the Web client and the like.
In order to manage audio and video sessions, the flow control layer and the access layer of each data center report the state of each data center to the configuration server, and the state is reported to the policy server by the configuration server, and the state can be used for adjusting the audio and video communication quality by the configuration server and the like.
In addition, the audio/video data transmission system may include an audio/video data transmission device, which may be specifically integrated in a server, for example, a relay server, and the following will take the example that the audio/video data transmission device is integrated in the relay server, where the relay server is mainly used to receive audio/video data sent by a sending client in audio/video sessions among a plurality of clients of different types, where the sending client is any one of the plurality of clients; acquiring a coding format required by each receiving client, wherein the receiving client is a client except for the sending client in the plurality of clients; then transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; respectively transmitting the transcoded audio and video data corresponding to each receiving client; and so on.
It should be noted that the scene schematic diagram of the audio/video data transmission system shown in fig. 3 is only an example, and the audio/video data transmission system and the scene described in the embodiment of the present invention are for more clearly illustrating the technical solution of the embodiment of the present invention, and do not form a limitation on the technical solution provided in the embodiment of the present invention.
The following are detailed below.
In this embodiment, the audio/video data transmission apparatus will be described from the perspective of the audio/video data transmission apparatus, and the audio/video data transmission apparatus may be specifically integrated in a network device such as a server or a gateway.
An audio and video data transmission method comprises the following steps: in audio and video sessions among a plurality of clients of different types, receiving audio and video data sent by a sending client, wherein the sending client is any one of the clients; acquiring coding formats required by all receiving clients, wherein the receiving clients are clients except for a sending client in a plurality of clients; transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; and respectively transmitting the transcoded audio and video data corresponding to each receiving client.
Referring to fig. 4, fig. 4 is a schematic flow chart of an audio/video data transmission method according to an embodiment of the present invention. The audio and video data transmission method can comprise the following steps:
in step S101, in an audio/video session between a plurality of clients of different types, audio/video data sent by a sending client is received, where the sending client is any one of the plurality of clients.
The plurality of clients of different types may include an SDK client, an applet client, a public address client, a browser client, a PSTN client, and the like, and may also include other types of clients, and specific contents are not limited herein.
The different types of the multiple clients may be that each client corresponds to a different type, for example, when the number of the multiple clients is 3, the 3 clients correspond to different types, such as an SDK client, a browser client, and a PSTN client, respectively, so that there are 3 different types of clients in total.
The plurality of clients of different types may also include a plurality of different types, and each different type may include one or more clients, for example, when there are 6 clients, the 6 clients may include 3 different types, such as an SDK client, a browser client, and a public number client, where the SDK client includes one, the browser client includes 2, and the public number client includes 3.
Multiple clients of different types can join the same audio and video session, and audio and video data interaction is carried out in the audio and video session. In a plurality of clients of different types in the same audio/video session, a sending client for sending audio/video data can be any one of the clients, and a receiving client for receiving the audio/video data can be any one of the clients. The audio and video data transmission device can receive and send audio and video data sent by the client through the audio and video session.
For example, in an audio-video session composed of an SDK client, an applet client, a public number client, a browser client, and a PSTN client, the browser client may be a sending client, and the SDK client, the applet client, the public number client, and the PSTN client may be receiving clients.
Before audio and video data transmission is performed through an audio and video session, the audio and video data transmission device may establish the audio and video session in advance, and in some embodiments, in the audio and video session between a plurality of clients of different types, before the step of receiving and sending audio and video data sent by the client, the audio and video data transmission method may further include:
(1) receiving session requests sent by a plurality of clients of different types, wherein the session requests carry session identifiers and client identifiers;
(2) changing account numbers for the clients according to the client identifications;
(3) and when determining that the audio/video session corresponding to the session identifier does not exist, establishing the audio/video session according to each client after the account is replaced.
Specifically, the audio/video data transmission device may receive session requests sent by a plurality of clients, for example, receive session requests sent by an SDK client, an applet client, a public client, a browser client, and the like, where the session requests may carry session identifiers, client identifiers, and the like, and may also carry other information, for example, an IP address of the client, and specific contents are not limited herein. The session identifier may be a name or a number of an audio/video session to be added, and the session identifier is used to uniquely identify the audio/video session, and may be composed of numbers, letters, and/or words, and the specific content is not limited herein. The client identifier may be a client name, an account number (e.g., a user name) or a number, etc., and the session identifier is used to uniquely identify the client, and may be composed of numbers, letters and/or words, etc., and the specific content is not limited herein.
In order to facilitate the audio and video data interaction of different types of clients, the audio and video data transmission device can replace accounts for the clients according to client identifications, accounts supported by the audio and video data transmission device and the like, so that the account types of the clients in audio and video sessions can be unified. For example, when a session request carrying a client identifier "abc" sent by a public number client is received, the account of the public number client may be replaced with "123456"; when a session request carrying a client identifier of "haha" sent by a browser client is received, the account number of the browser client can be changed to "456789".
The audio and video data transmission device can judge whether an audio and video session corresponding to the session identifier exists, and if the audio and video session corresponding to the session identifier exists, the client can join the audio and video session; if the audio/video session corresponding to the session identifier does not exist, the audio/video session can be established according to each client after the account is replaced. For example, the audio/video data transmission device may allocate a relay IP to the client, and send the relay IP to the client, so that the client may establish a secure connection with the audio/video data transmission device according to the relay IP. Therefore, the network connection established by the audio and video data transmission device can be added into the audio and video session.
In some embodiments, after the step of receiving the audio and video data sent by the sending client in the audio and video session between the multiple clients of different types, the audio and video data transmission method may further include:
acquiring the type of audio and video data; and when the audio and video data need to be decrypted according to the type of the audio and video data, decrypting the audio and video data to obtain the decrypted audio and video data.
Specifically, when the sending client sends the audio and video data through the audio and video session, the audio and video data transmission device may call a packet receiving thread to receive the audio and video data sent by the sending client, and write the received audio and video data into a packet receiving buffer. At this time, the audio/video data transmission device may call the main thread to read the audio/video data from the packet receiving buffer and obtain the type of the audio/video data, where the type of the audio/video data may include audio/video data sent by an SDK client, audio/video data sent by an applet client, audio/video data sent by a PSTN client, or audio/video data sent by a Web client. The audio and video data transmission device can judge whether the audio and video data need to be decrypted according to the type of the audio and video data, and can decrypt the audio and video data to obtain the decrypted audio and video data when the audio and video data need to be decrypted according to the type of the audio and video data. When the audio and video data are determined not to need to be decrypted according to the type of the audio and video data, the audio and video data do not need to be decrypted.
For example, when the audio and video data is sent by the SDK client, the audio and video data does not need to be decrypted; when the audio and video data is sent by clients such as a public client, a browser client and the like, the audio and video data needs to be decrypted to obtain the decrypted audio and video data.
In some embodiments, the step of obtaining the type of the audio/video data, and when it is determined that the audio/video data needs to be decrypted according to the type of the audio/video data, decrypting the audio/video data to obtain the decrypted audio/video data may include:
(1) calling a main thread to extract a packet header from the audio and video data, and determining the type of the audio and video data according to a type field in the packet header;
(2) when the audio and video data need to be decrypted is determined according to the type of the audio and video data, the audio and video data are written into a buffer area to be decrypted through a main thread;
(3) and calling a decryption thread to read the audio and video data from the buffer area to be decrypted, decrypting the audio and video data according to a preset decryption key, and writing the decrypted audio and video data into the decrypted buffer area.
Specifically, the audio/video data transmission device may invoke the main thread to read audio/video data from a packet receiving buffer, analyze the read audio/video data, extract a packet header from the audio/video data, and extract content of a type field in the packet header, where the type field in the packet header is used to store the type of the audio/video data, and at this time, may determine the type of the audio/video data according to the content of the type field in the packet header, for example, the audio/video data sent by an SDK client, or the audio/video data sent by clients such as an applet client, a PSTN client, and a Web client.
The audio and video data transmission device can call a main thread to judge whether the audio and video data need to be decrypted according to the type of the audio and video data, and when the audio and video data need to be decrypted is determined according to the type of the audio and video data, the audio and video data transmission device can write the audio and video data into a buffer area to be decrypted through the main thread; when the audio and video data are determined not to need decryption according to the type of the audio and video data, the audio and video data transmission device can write the audio and video data into a buffer area to be transcoded through the main thread.
When decryption is needed, the audio and video data transmission device can call a decryption thread to read audio and video data from the buffer area to be decrypted, decrypt the audio and video data according to a preset decryption key, and write the decrypted audio and video data into the decrypted buffer area, wherein the preset decryption key can be the decryption key negotiated when the audio and video data transmission device establishes safe connection with the client.
It should be noted that when the audio and video data are decrypted, the audio data and the video data in the audio and video data can be decrypted respectively, and when a plurality of audio data and a plurality of video data need to be decrypted, the audio and video data transmission device can call a plurality of decryption threads to decrypt the plurality of audio data and the plurality of video data in parallel according to actual needs.
It should be noted that the main thread may be configured according to the requirements of the service scenario, and the number of the sub-threads corresponding to the main thread are configured, for example, the decryption thread, the transcoding thread, the encryption thread, and the like may all be the sub-threads corresponding to the main thread, and the number of the transcoding thread may be started according to the requirements of the function, the load, and the like, so as to fully utilize the server resources.
Various threads such as a packet receiving thread, a decryption thread, a transcoding thread, an encryption thread, a packet sending thread and the like related to the embodiment of the invention can be replaced by a process or a server and the like, for example, an audio and video data transmission device can call the decryption process to decrypt audio and video data, and the decryption process can comprise one or more decryption threads and the like; for another example, the audio/video data transmission device may invoke the transcoding server to transcode the audio/video data, and the like, and may be flexibly set according to actual needs, and specific content is not limited here.
The various buffers related to the embodiment of the present invention, such as a packet receiving buffer, a buffer to be decrypted, a decrypted buffer, a buffer to be transcoded, a buffer to be encrypted, an encrypted buffer, and a packet sending buffer, may be a ring buffer queue (e.g., a single-read single-write lock-free ring buffer queue), or a shared memory or other storage space, and may be flexibly set according to actual needs, and specific contents are not limited herein. Audio and video data are exchanged between various sub-threads and a main thread through various buffer areas without mutual influence, the data format of the audio and video data can be 'data length + other optional appointed fields + data', and the like, the various sub-threads and the main thread can read the data in the corresponding buffer areas in real time or at preset time intervals, and the specific content is not limited here.
In step S102, the encoding format required by each receiving client is obtained, and the receiving client is a client other than the sending client among the plurality of clients.
In the same audio-video session, when the sending client sends the audio-video data, the audio-video data transmission device can acquire the coding formats required by all receiving clients except the sending client in the audio-video session. For example, before joining an audio/video session, a client (including a sending client and a receiving client) may send a data transmission capability negotiation request to an audio/video data transmission device through a signaling server, where the data transmission capability negotiation request may carry an audio data format, a video data format, and the like supported by the client, and the audio data format and the video data format are encoding formats of audio/video data. When a data transmission capability negotiation request is received, the audio and video data transmission device can store an audio data format, a video data format and the like supported by a client, determine a coding format (namely a data format) when audio and video data transmission is carried out with the client according to the audio data format and the video data format supported by the client and the audio data format and the video data format supported by the audio and video data transmission device, and send a data transmission capability negotiation response to the client through a signaling server, wherein the data transmission capability negotiation response comprises the coding format and the like when the audio and video data transmission is carried out with the client.
Therefore, the audio/video data transmission device may send a coding format obtaining request to each receiving client and receive a coding format returned by each receiving client, or the audio/video data transmission device may obtain a coding format and the like required by each receiving client from a pre-stored database of coding formats required by each client, where the coding formats include an audio coding format, a video coding format and the like.
In step S103, the audio/video data is transcoded according to the coding format required by each receiving client, so as to obtain transcoded audio/video data corresponding to each receiving client.
After the coding formats required by the receiving clients are obtained, the audio and video data transmission device can judge whether the audio and video data need to be transcoded according to the coding formats of the audio and video data and the coding formats required by the receiving clients. The encoding formats of the Audio and video data include an Audio encoding format and a video encoding format, wherein the Audio encoding format may include Advanced Audio Coding (AAC), OPUS, g.711, g.719, and silk, and the like, and the video encoding format may include h264, h265, vp8, vp9, and the like.
Because different types of clients exist in the audio and video session, the encoding formats required by the clients may be different, at this time, when the encoding formats of the audio and video data are different from the encoding formats required by the receiving clients, it is determined that the audio and video data need to be transcoded, and the audio and video data transmission device can transcode the audio and video data respectively according to the encoding formats required by the receiving clients to obtain the transcoded audio and video data corresponding to the receiving clients. For example, the audio data may be transcoded according to the audio coding format required by each receiving client, so as to obtain transcoded audio data corresponding to each receiving client; and transcoding the video data according to the video coding format required by each receiving client to obtain the transcoded video data corresponding to each receiving client.
It can be understood that when the encoding format required by a certain receiving client in each receiving client is consistent with the encoding format of the audio and video data sent by the sending client, the audio and video data corresponding to the certain receiving client does not need to be transcoded.
For example, when the audio encoding format is g.711 and the video encoding format is h264 in the audio and video data of the sending client, the audio encoding format required by the receiving client a is AAC and the video encoding format required by the receiving client a is h 265; the audio encoding format required by the receiving client B is OPUS and the video encoding format required is vp 9. At this time, the audio and video data transmission device may transcode the audio coding format g.711 into AAC, and transcode the video coding format h264 into h265, so as to obtain transcoded audio data AAC and video data h265 corresponding to the receiving client a; and transcoding g.711 in the audio coding format into OPUS, and transcoding h264 in the video coding format into vp9, so as to obtain transcoded audio data OPUS and video data vp9 and the like corresponding to the receiving client B.
In some embodiments, the step of obtaining the encoding format required by each receiving client, and transcoding the audio/video data according to the encoding format required by each receiving client, to obtain transcoded audio/video data corresponding to each receiving client may include:
(1) acquiring attribute information of each receiving client, and determining a coding format required by each receiving client according to the attribute information;
(2) calling the main thread to read the decrypted audio and video data, and writing the decrypted audio and video data into a buffer area to be code-converted;
(3) and calling an audio and video transcoding thread to read the decrypted audio and video data from the buffer area to be transcoded, and transcoding the decrypted audio and video data respectively according to the coding format required by each receiving client to obtain the transcoded audio and video data corresponding to each receiving client.
Specifically, the audio/video data transmission device may obtain attribute information of each receiving client, where the attribute information may include a type of the receiving client, an audio coding format and a video coding format supported by the receiving client, and the like, and for example, the audio/video data transmission device may send an attribute information obtaining request to each receiving client, and receive attribute information returned by each receiving client; or, the audio/video data transmission device may pre-store the attribute information of the client when establishing connection with the receiving client, and may obtain the attribute information of each receiving client from the database storing the attribute information of the client when the attribute information needs to be obtained. The audio and video data transmission device can determine the coding format required by each receiving client according to the attribute information, and at the moment, if the audio and video data are decrypted, the main thread can be called to read the decrypted audio and video data from the decrypted buffer area and write the decrypted audio and video data into the buffer area to be code-converted. And calling an audio and video transcoding thread to read the decrypted audio and video data from the buffer area to be transcoded, and transcoding the decrypted audio and video data respectively according to the coding format required by each receiving client to obtain the transcoded audio and video data corresponding to each receiving client.
It can be understood that, if the audio and video data does not need to be decrypted, the audio and video data transmission device may directly invoke the audio and video transcoding thread to read the audio and video data from the buffer area to be transcoded, and transcode the audio and video data respectively according to the coding format required by each receiving client, so as to obtain the transcoded audio and video data corresponding to each receiving client.
In some embodiments, the step of calling the main thread to read the decrypted audio and video data, writing the decrypted audio and video data into the buffer area to be transcoded, calling the audio and video transcoding thread to read the decrypted audio and video data from the buffer area to be transcoded, and transcoding the decrypted audio and video data according to the coding format required by each receiving client, to obtain the transcoded audio and video data corresponding to each receiving client may include:
(a) calling a main thread to read the decrypted audio and video data, and writing the audio data in the decrypted audio and video data into an audio buffer area to be transcoded;
(b) writing the video data in the decrypted audio and video data into a video buffer area to be transcoded through the main thread;
(c) calling an audio transcoding thread to read audio data from an audio buffer area to be transcoded, and transcoding the audio data according to the audio coding format required by each receiving client to obtain transcoded audio data corresponding to each receiving client;
(d) calling a video transcoding thread to read video data from a video buffer area to be transcoded, and transcoding the video data according to a video coding format required by each receiving client to obtain transcoded video data corresponding to each receiving client;
(e) and setting the transcoded audio data and the transcoded video data corresponding to each receiving client as the transcoded audio/video data corresponding to each receiving client.
The audio and video data transmission device can respectively process the audio data and the video data in the audio and video data, and specifically, the audio and video data transmission device can call a main thread to read the decrypted audio and video data from the decrypted buffer area, write the audio data in the decrypted audio and video data into an audio buffer area to be transcoded, and write the video data in the decrypted audio and video data into a video buffer area to be transcoded. At this time, the audio and video data transmission device can call an audio transcoding thread to read audio data from the audio buffer area to be transcoded, and transcode the audio data according to the audio coding format required by each receiving client, so as to obtain transcoded audio data corresponding to each receiving client. And the audio and video data transmission device can call a video transcoding thread to read video data from a video buffer area to be transcoded, and transcode the video data according to the video coding format required by each receiving client to obtain the transcoded video data corresponding to each receiving client.
The transcoded audio data corresponding to each receiving client and the transcoded video data corresponding to each receiving client are the transcoded audio/video data corresponding to each receiving client. The audio and video data transmission device can independently realize the heavier functions such as decryption, audio transcoding, video transcoding and the like according to the function and load difference, can flexibly configure the required functions according to the service requirements, and can configure the resource allocation of each functional block, thereby realizing different types of clients such as SDK clients, small program clients, H5 clients, PSTN clients and the like without being limited by the respective audio and video coding and decoding difference and the like, solving the requirements of audio and video data audio coding and decoding formats, video coding and decoding formats and the like required by different clients and realizing the fusion and intercommunication of audio and video data between the different types of clients.
It should be noted that transcoding audio data and video data may share a thread, for example, a transcoding thread, that is, the audio and video data transmission apparatus may invoke the transcoding thread to transcode audio data, and may also invoke the transcoding thread to transcode video data. For example, when audio data needs to be transcoded, the audio and video data transmission device may call a transcoding thread to read the audio data from a buffer area to be transcoded, and transcode the audio data; when the video data needs to be transcoded, the audio and video data transmission device can call a transcoding thread to read the video data from the buffer area to be transcoded and transcode the video data.
In step S104, the transcoded audio/video data corresponding to each receiving client is transmitted to each receiving client, respectively.
After the transcoded audio/video data corresponding to each receiving client is obtained, the audio/video data transmission device can transmit the transcoded audio/video data corresponding to each receiving client respectively.
In some embodiments, before the step of transmitting the transcoded audio/video data corresponding to each receiving client, the audio/video data transmission method may further include:
and when the transcoded audio and video data needs to be encrypted, encrypting the transcoded audio and video data to obtain the encrypted audio and video data.
The step of transmitting the transcoded audio/video data corresponding to each receiving client respectively may include: and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
Specifically, the audio/video data transmission device may determine whether the transcoded audio/video data needs to be encrypted, for example, may determine whether the transcoded audio/video data needs to be encrypted according to the type of the receiving client, and when the receiving client is an SDK client, the audio/video data does not need to be encrypted; when the receiving client is a public client, a browser client or the like, the audio and video data may need to be encrypted. For another example, whether the transcoded audio and video data needs to be encrypted can be judged according to the type of the audio and video data. When the audio and video data after transcoding is determined to need to be encrypted, the audio and video data after transcoding can be encrypted to obtain the encrypted audio and video data.
In some embodiments, when the transcoded audio/video data needs to be encrypted, the step of encrypting the transcoded audio/video data to obtain encrypted audio/video data, and the step of transmitting the encrypted audio/video data corresponding to each receiving client respectively may include:
(a) calling a main thread to read the transcoded audio and video data and acquiring the type of the transcoded audio and video data;
(b) when determining that the transcoded audio and video data needs to be encrypted according to the type of the transcoded audio and video data, writing the transcoded audio and video data into a buffer area to be encrypted;
(c) calling an encryption thread to read transcoded audio and video data from a buffer area to be encrypted, encrypting the transcoded audio and video data according to a preset encryption key, and writing the encrypted audio and video data into an encrypted buffer area;
(d) calling a main thread to read the encrypted audio and video data from the encrypted buffer area and writing the encrypted audio and video data into a packet sending buffer area;
(e) and calling a packet sending thread to read the encrypted audio and video data from the packet sending buffer area, and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
Specifically, the audio/video data transmission device may call the main thread to read transcoded audio/video data from the transcoded buffer region (for example, call the main thread to read transcoded audio data from the audio transcoded buffer region and read transcoded video data from the video transcoded buffer region), obtain the type of the transcoded audio/video data, and determine whether to encrypt the transcoded audio/video data according to the type of the transcoded audio/video data. For example, the transcoded audio/video data may be analyzed, a packet header is extracted from the transcoded audio/video data, and a type field for storing the type of the audio/video data in the packet header is extracted, and at this time, the type of the audio/video data may be determined according to the content of the type field in the packet header, for example, the audio/video data received by an SDK client, or the audio/video data received by clients such as an applet client, a PSTN client, and a Web client.
When the transcoded audio and video data needs to be encrypted according to the type of the transcoded audio and video data, the audio and video data transmission device can write the transcoded audio and video data into a packet sending buffer area; when the transcoded audio and video data needs to be encrypted according to the type of the transcoded audio and video data, the audio and video data transmission device can write the transcoded audio and video data into a buffer area to be encrypted.
When encryption is needed, the audio and video data transmission device can call an encryption thread to read transcoded audio and video data from a buffer area to be encrypted, encrypt the transcoded audio and video data according to a preset encryption key, and write the encrypted audio and video data into the encrypted buffer area; the preset encryption key may be an encryption key negotiated when the audio/video data transmission device establishes a secure connection with the client.
It should be noted that when the audio and video data are encrypted, the audio data and the video data in the audio and video data can be respectively encrypted, and when a plurality of audio data and a plurality of video data need to be encrypted, the audio and video data transmission device can call a plurality of encryption threads to encrypt the plurality of audio data and the plurality of video data in parallel according to actual needs.
At this time, the audio/video data transmission device can call the main thread to read the encrypted audio/video data from the encrypted buffer area and write the encrypted audio/video data into the packet sending buffer area. And then calling a packet sending thread to read the encrypted audio and video data from the packet sending buffer area, and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
The audio and video data transmission device can independently perform heavier functions such as security encryption and decryption, audio transcoding, video transcoding, data transmission and the like according to the function and load difference, can flexibly configure required functions according to business requirements, can configure resource allocation of each functional block, and can efficiently utilize resources; the method and the system realize different types of clients such as the SDK client, the applet client, the PSTN client, the Web client and the like without being limited by respective audio and video coding and decoding differences, safety transmission requirement differences and the like, solve the requirements of audio and video data audio coding and decoding formats, video coding and decoding formats, transmission safety and the like required by different clients and realize the fusion and intercommunication of audio and video data among different types of clients.
It should be noted that, the encryption of the audio and video data and the encryption of the audio and video data may share one thread, for example, an encryption and decryption thread, that is, the audio and video data transmission device may call the encryption and decryption thread to encrypt the audio and video data, and may also call the encryption and decryption thread to encrypt the audio and video data.
For example, when the audio and video data needs to be decrypted, the audio and video data transmission device may call the encryption and decryption thread to read the audio and video data from the buffer to be encrypted and decrypted, and extract a preset field from the audio and video data, where the preset field is used to store an encryption instruction or a decryption instruction, so that the encryption and decryption thread may encrypt the audio and video data according to the encryption instruction, or may decrypt the audio and video data according to the decryption instruction. When the fact that decryption operation is needed is determined according to a decryption instruction in a preset field, the audio and video data transmission device can decrypt the audio and video data through the encryption and decryption thread, the decrypted audio and video data are written into the encryption and decryption buffer area, the decrypted audio and video data can comprise a field used for storing decrypted information, so that after the main thread reads the audio and video data from the encryption and decryption buffer area, the field used for storing the decrypted information can be extracted from the audio and video data, and the audio and video data are determined to be the decrypted audio and video data according to the decrypted information.
When the audio and video data needs to be encrypted, the audio and video data transmission device can call an encryption and decryption thread to read the audio and video data from a buffer area to be encrypted and decrypted and extract a preset field from the audio and video data, when the encryption operation needs to be performed according to an encryption instruction in the preset field, the audio and video data transmission device can encrypt the audio and video data through the encryption and decryption thread and write the encrypted audio and video data into the encryption and decryption buffer area, the encrypted audio and video data can comprise a field for storing encrypted information, so that after the main thread reads the audio and video data from the encryption and decryption buffer area, the field for storing the encrypted information can be extracted from the audio and video data, and the audio and video data are determined to be the encrypted audio and video data according to the encrypted information.
As can be seen from the above, the embodiment of the present invention can receive the audio and video data sent by the sending client in the audio and video session among the multiple clients of different types, where the sending client is any one of the multiple clients; acquiring a coding format required by each receiving client, wherein the receiving client is a client except for the sending client in the plurality of clients; then transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; and respectively transmitting the transcoded audio and video data corresponding to each receiving client. According to the scheme, multiple clients of different types can join the same audio and video session, audio and video data are transcoded in the audio and video session according to the coding formats required by the receiving clients, and the transcoded audio and video data are transmitted to the receiving clients respectively, so that the audio and video data interaction of the clients of different types in the same audio and video session is realized, the limitation that the same client can join the same audio and video session to transmit the audio and video data in the prior art is overcome, and the flexibility of audio and video data transmission is greatly improved.
The method described in the above embodiments is further illustrated in detail by way of example.
In this embodiment, an audio/video data transmission device is taken as an example of a transit server, and an audio/video session is established among a browser client, an SDK client, and an applet client, where the browser client is a client that sends audio/video data, and the SDK client and the applet client are both clients that receive the audio/video data. In the embodiment of the invention, the browser client, the SDK client, the applet client and the like can decode the audio and video data sent by the browser client according to actual needs and can encrypt the received audio and video data according to actual needs. For convenience of illustration, the following will take as an example that the audio and video data sent by the SDK client does not need to be decoded, and the received audio and video data does not need to be encrypted; for example, the audio and video data sent by the applet client and the browser client need to be decoded, and the received audio and video data need to be encrypted.
It should be noted that, there may be not only these 3 types of clients in an audio-video session, but this embodiment is only an example given for convenience of description, and should not be understood as a limitation to the types and numbers of clients in an audio-video session, and the transmission process of audio-video data is similar regardless of how many clients and how many types of clients exist, and can be understood according to this example.
Referring to fig. 5, fig. 5 is another schematic flow chart of the audio/video data transmission method according to the embodiment of the present invention. The method flow can comprise the following steps:
s201, the browser client sends audio and video data to the transit server.
In the same audio and video session comprising the browser client, the SDK client and the applet client, the integration and intercommunication of audio and video data can be carried out among the clients, when the browser client needs to send the audio and video data to the SDK client and the applet client, the browser client firstly sends the audio and video data to the transfer server through the audio and video session, and the audio and video data sent by the browser client can be called as uplink audio and video data.
At this time, the transit server may call a packet receiving thread to receive the audio and video data sent by the browser client, and write the received audio and video data into a packet receiving buffer through the packet receiving thread, for example, as shown in fig. 6.
S202, the transit server decrypts and transcodes the audio and video data.
After receiving the audio and video data sent by the browser client, the transfer server may invoke the main thread to read the audio and video data from the packet receiving buffer, for example, as shown in fig. 6, the read audio and video data may be analyzed by the main thread, a packet header may be extracted from the audio and video data, and a content of a type field in the packet header may be extracted, where the type field in the packet header is used to store a type of the audio and video data, and at this time, a type of the audio and video data may be determined according to the content of the type field in the packet header. When the audio and video data sent by the browser client side is determined according to the type of the audio and video data, the audio and video data can be judged to need to be decrypted, and the transit server can write the audio and video data into a buffer area to be encrypted and decrypted through a main thread, so that the transit server can call an encryption and decryption thread to read the audio and video data from the buffer area to be encrypted and decrypted, wherein the buffer area to be encrypted and decrypted can store the audio and video data to be decrypted and can also store the audio and video data to be encrypted; the encryption and decryption thread can decrypt the audio and video data and can also encrypt the audio and video data.
When the audio and video data need to be decrypted, the transfer server can call the encryption and decryption thread to read the audio and video data from the buffer area to be encrypted and decrypted, and extract a preset field from the audio and video data, wherein the preset field is used for storing an encryption instruction or a decryption instruction, so that the encryption and decryption thread can encrypt the audio and video data according to the encryption instruction or decrypt the audio and video data according to the decryption instruction. When the fact that decryption operation is needed is determined according to the decryption instruction in the preset field, the transfer server can decrypt the audio and video data through the encryption and decryption thread according to a preset decryption key, and the preset decryption key can be a decryption key negotiated when the audio and video data transmission device is in safe connection with the client. And then writing the decrypted audio and video data into the encrypted and decrypted buffer area, wherein the decrypted audio and video data can comprise a field for storing decrypted information, so that after the main thread reads the audio and video data from the encrypted and decrypted buffer area, the field for storing the decrypted information can be extracted from the audio and video data, and the audio and video data is determined to be the decrypted audio and video data according to the decrypted information.
It should be noted that the encryption and decryption threads in fig. 6 may include an encryption and decryption thread 1 to an encryption and decryption thread n, where a value of n may be set according to actual needs, a specific value is not limited here, and the relay server may call a preset number of encryption and decryption threads to decrypt the audio and video data according to actual needs.
After the audio and video data are decrypted, the transfer server can acquire coding formats required by the SDK client and the applet client, wherein the coding formats comprise an audio coding format and a video coding format. When the audio and video data are determined to need transcoding according to the coding formats required by the SDK client and the applet client, the transfer server can call the main thread to read the audio and video data from the encrypted and decrypted buffer area, and according to a field used for storing decrypted information in the audio and video data, the transfer server knows that the audio and video data are the decrypted audio and video data, and at the moment, the decrypted audio and video data can be written into a buffer area to be transcoded, for example, the audio data in the decrypted audio and video data are written into an audio buffer area to be transcoded, and the video data in the decrypted audio and video data are written into a video buffer area to be transcoded.
The transfer server can call the audio transcoding thread 1 to read audio data from the audio buffer area to be transcoded, transcode the audio data according to the audio coding format required by the SDK client, obtain transcoded audio data corresponding to the SDK client, and write the transcoded audio data into the audio buffer area which has been transcoded. And calling an audio transcoding thread n to read audio data from the audio buffer area to be transcoded, transcoding the audio data according to the audio coding format required by the small program client to obtain transcoded audio data corresponding to the small program client, and writing the transcoded audio data into the audio buffer area which is transcoded.
The transfer server can call the video transcoding thread 1 to read video data from the video buffer area to be transcoded, transcode the video data according to the video coding format required by the SDK client, obtain transcoded video data corresponding to the SDK client, and write the transcoded video data into the video buffer area which has been transcoded. And calling a video transcoding thread n to read video data from the video buffer area to be transcoded, transcoding the video data according to a video coding format required by the small program client to obtain transcoded video data corresponding to the small program client, and writing the transcoded video data into the video buffer area which is transcoded.
It should be noted that the audio transcoding thread in fig. 6 may include an audio transcoding thread 1 to an audio transcoding thread n, where a value of n may be set according to actual needs, and a specific value is not limited herein. The video transcoding threads can also comprise a video transcoding thread 1 to a video transcoding thread n, wherein the value of n can be set according to actual needs, the specific value is not limited at this point, the transfer server can call a preset number of audio transcoding threads to transcode the audio data according to the actual needs, and can call the preset number of video transcoding threads to transcode the video data according to the actual needs.
And S203, the transfer server sends the transcoded audio and video data to the SDK client.
After the transcoded audio and video data corresponding to the SDK client is obtained, the transfer server can call the main thread to read the transcoded audio data corresponding to the SDK client from the audio transcoded buffer area and read the transcoded video data corresponding to the SDK client from the video transcoded buffer area, and determine that the transcoded audio and video data corresponding to the SDK client does not need to be encrypted according to the type of the audio and video data required by the SDK client, and at the moment, the transfer server can write the transcoded audio data and video data corresponding to the SDK client into the packet sending buffer area through the main thread. Then, the relay server may call a packet sending thread to read the transcoded audio/video data corresponding to the SDK client from the packet sending buffer, and send the transcoded audio/video data corresponding to the SDK client through the packet sending thread, where the audio/video data sent by the packet sending thread is the downlink audio/video data, for example, as shown in fig. 6.
And S204, encrypting the transcoded audio and video data by the transfer server.
After the transcoded audio and video data corresponding to the small program client is obtained, the transfer server can call the main thread to read the transcoded audio data corresponding to the small program client from the audio transcoded buffer area and read the transcoded video data corresponding to the small program client from the video transcoded buffer area, and determine that the transcoded audio and video data corresponding to the small program client needs to be encrypted according to the type of the audio and video data required by the small program client, and at the moment, the transfer server can call the main thread to write the transcoded audio and video data (including the audio data and the video data) corresponding to the small program client into a buffer area to be encrypted and decrypted.
When the audio and video data needs to be encrypted, the transfer server can call an encryption and decryption thread to read transcoded audio and video data corresponding to the small program client from the buffer area to be encrypted and decrypted and extract a preset field from the audio and video data, when the encryption operation is determined to be needed according to the encryption instruction in the preset field, the transit server can encrypt the audio and video data through the encryption and decryption thread, for example, the audio data and the video data in the audio and video data are respectively encrypted, the encrypted audio and video data are written into the encrypted and decrypted buffer area, the encrypted audio-video data may include a field for storing encrypted information, so that after the main thread reads the audio-video data from the encrypted and decrypted buffer, the field for storing the encrypted information can be extracted from the audio and video data, and the audio and video data is determined to be the encrypted audio and video data according to the encrypted information.
And S205, the transit server sends the encrypted audio and video data to the applet client.
The transfer server can call the main thread to read the audio and video data corresponding to the small program client from the encrypted and decrypted buffer area and write the audio and video data into the packet sending buffer area. Then, the transit server may call a packet sending thread to read the encrypted audio and video data corresponding to the applet client from the packet sending buffer, and send the encrypted audio and video data corresponding to the applet client through the packet sending thread, where the audio and video data sent by the packet sending thread is the downlink audio and video data, for example, as shown in fig. 6.
According to the embodiment of the invention, the transfer server can independently perform the functions such as encryption, decryption, audio transcoding, video transcoding, data transmission and the like with heavier ratios according to the functions such as encryption, decryption, transcoding and the like, and the load difference of each thread and the like, and can flexibly configure each thread to execute the required functions according to the service requirements, so that the resources can be efficiently utilized; the client sides of different types are not limited by respective audio and video coding and decoding differences and safety transmission requirement differences, the requirements of audio and video data on audio coding and decoding formats, video coding and decoding formats, transmission safety and the like required by the client sides of different types are met, the fusion and intercommunication of audio and video data among the client sides of different types are realized, and the flexibility of audio and video data transmission is greatly improved. .
In order to better implement the audio and video data transmission method provided by the embodiment of the invention, the embodiment of the invention also provides a device based on the audio and video data transmission method. The meaning of the nouns is the same as that in the audio and video data transmission method, and specific implementation details can refer to the description in the method embodiment.
Referring to fig. 7, fig. 7 is a schematic structural diagram of an audio/video data transmission apparatus according to an embodiment of the present invention, where the audio/video data transmission apparatus may include a first receiving unit 301, an obtaining unit 302, a transcoding unit 303, a transmission unit 304, and the like.
The first receiving unit 301 is configured to receive, in an audio/video session between multiple clients of different types, audio/video data sent by a sending client, where the sending client is any one of the multiple clients.
The plurality of clients of different types may include an SDK client, an applet client, a public address client, a browser client, a PSTN client, and the like, and may also include other types of clients, and specific contents are not limited herein.
The different types of the multiple clients may be that each client corresponds to a different type, for example, when the number of the multiple clients is 3, the 3 clients correspond to different types, such as an SDK client, a browser client, and a PSTN client, respectively, so that there are 3 different types of clients in total.
The plurality of clients of different types may also include a plurality of different types, and each different type may include one or more clients, for example, when there are 6 clients, the 6 clients may include 3 different types, such as an SDK client, a browser client, and a public number client, where the SDK client includes one, the browser client includes 2, and the public number client includes 3.
Multiple clients of different types can join the same audio and video session, and audio and video data interaction is carried out in the audio and video session. In a plurality of clients of different types in the same audio/video session, a sending client for sending audio/video data can be any one of the clients, and a receiving client for receiving the audio/video data can be any one of the clients. The first receiving unit 301 may receive audio and video data sent by the sending client through an audio and video session.
For example, in an audio-video session composed of an SDK client, an applet client, a public number client, a browser client, and a PSTN client, the browser client may be a sending client, and the SDK client, the applet client, the public number client, and the PSTN client may be receiving clients.
Before audio and video data transmission is performed through an audio and video session, the audio and video data transmission device may establish the audio and video session in advance, and in some embodiments, as shown in fig. 10, the audio and video data transmission device may further include a second receiving unit 307, a replacing unit 308, an establishing unit 309, and the like, which may specifically be as follows:
a second receiving unit 307, configured to receive session requests sent by multiple clients of different types, where the session requests carry session identifiers and client identifiers;
a replacing unit 308, configured to replace an account for each client according to the client identifier;
the establishing unit 309 is configured to establish an audio/video session according to each client after the account is replaced when it is determined that the audio/video session corresponding to the session identifier does not exist.
Specifically, the second receiving unit 307 may receive session requests sent by a plurality of clients, for example, receive session requests sent by an SDK client, an applet client, a public client, a browser client, and the like, where the session requests may carry a session identifier, a client identifier, and other information, for example, an IP address of the client, and the specific content is not limited herein. The session identifier may be a name or a number of an audio/video session to be added, and the session identifier is used to uniquely identify the audio/video session, and may be composed of numbers, letters, and/or words, and the specific content is not limited herein. The client identifier may be a client name, an account number (e.g., a user name) or a number, etc., and the session identifier is used to uniquely identify the client, and may be composed of numbers, letters and/or words, etc., and the specific content is not limited herein.
In order to facilitate the audio and video data interaction of different types of clients, the replacing unit 308 may replace an account for each client according to the client identifier, the account supported by the audio and video data transmission device, and the like, so as to unify the account types of each client in the audio and video session. For example, when a session request carrying a client identifier "abc" sent by a public number client is received, the account of the public number client may be replaced with "123456"; when a session request carrying a client identifier of "haha" sent by a browser client is received, the account number of the browser client can be changed to "456789".
The establishing unit 309 may determine whether an audio/video session corresponding to the session identifier exists, and if an audio/video session corresponding to the session identifier exists, the client may join the audio/video session; if the audio/video session corresponding to the session identifier does not exist, the audio/video session can be established according to each client after the account is replaced. For example, the audio/video data transmission device may allocate a relay IP to the client, and send the relay IP to the client, so that the client may establish a secure connection with the audio/video data transmission device according to the relay IP. Therefore, the network connection established by the audio and video data transmission device can be added into the audio and video session.
In some embodiments, as shown in fig. 8, the audio-video data transmission apparatus may further include: the decryption unit 305 is configured to acquire the type of the audio/video data, and when it is determined that the audio/video data needs to be decrypted according to the type of the audio/video data, decrypt the audio/video data to obtain decrypted audio/video data.
Specifically, when the sending client sends the audio and video data through the audio and video session, the first receiving unit 301 may call a packet receiving thread to receive the audio and video data sent by the sending client, and write the received audio and video data into a packet receiving buffer. At this time, the decryption unit 305 may invoke the main thread to read the audio and video data from the packet receiving buffer and obtain the type of the audio and video data, where the type of the audio and video data may include audio and video data sent by an SDK client, audio and video data sent by an applet client, audio and video data sent by a PSTN client, or audio and video data sent by a Web client. The decryption unit 305 may judge whether the audio and video data needs to be decrypted according to the type of the audio and video data, and when it is determined that the audio and video data needs to be decrypted according to the type of the audio and video data, the decryption unit 305 may decrypt the audio and video data to obtain decrypted audio and video data. When the audio and video data are determined not to need to be decrypted according to the type of the audio and video data, the audio and video data do not need to be decrypted.
For example, when the audio and video data is sent by the SDK client, the audio and video data does not need to be decrypted; when the audio and video data is sent by clients such as a public client, a browser client and the like, the audio and video data needs to be decrypted to obtain the decrypted audio and video data.
In some embodiments, the decryption unit 305 may be specifically configured to:
calling a main thread to extract a packet header from the audio and video data, and determining the type of the audio and video data according to a type field in the packet header;
when the audio and video data need to be decrypted is determined according to the type of the audio and video data, the audio and video data are written into a buffer area to be decrypted through a main thread;
and calling a decryption thread to read the audio and video data from the buffer area to be decrypted, decrypting the audio and video data according to a preset decryption key, and writing the decrypted audio and video data into the decrypted buffer area.
Specifically, the decryption unit 305 may invoke the main thread to read the audio and video data from the packet receiving buffer, analyze the read audio and video data, extract the packet header from the audio and video data, and extract the content of the type field in the packet header, where the type field in the packet header is used to store the type of the audio and video data, and at this time, may determine the type of the audio and video data according to the content of the type field in the packet header, for example, the audio and video data sent by the SDK client, or the audio and video data sent by the clients such as the applet client, the PSTN client, and the Web client.
The decryption unit 305 may call a main thread to determine whether the audio and video data needs to be decrypted according to the type of the audio and video data, and when it is determined that the audio and video data needs to be decrypted according to the type of the audio and video data, the audio and video data transmission device may write the audio and video data into a buffer to be decrypted through the main thread; when the audio and video data are determined not to need decryption according to the type of the audio and video data, the audio and video data transmission device can write the audio and video data into a buffer area to be transcoded through the main thread.
When decryption is required, the decryption unit 305 may invoke a decryption thread to read the audio and video data from the buffer to be decrypted, decrypt the audio and video data according to a preset decryption key, and write the decrypted audio and video data into the decrypted buffer, where the preset decryption key may be a decryption key negotiated when the audio and video data transmission device establishes a secure connection with the client.
It should be noted that when the audio and video data are decrypted, the audio data and the video data in the audio and video data can be decrypted respectively, and when a plurality of audio data and a plurality of video data need to be decrypted, the audio and video data transmission device can call a plurality of decryption threads to decrypt the plurality of audio data and the plurality of video data in parallel according to actual needs.
It should be noted that the main thread may be configured according to the requirements of the service scenario, and the number of the sub-threads corresponding to the main thread are configured, for example, the decryption thread, the transcoding thread, the encryption thread, and the like may all be the sub-threads corresponding to the main thread, and the number of the transcoding thread may be started according to the requirements of the function, the load, and the like, so as to fully utilize the server resources.
Various threads such as a packet receiving thread, a decryption thread, a transcoding thread, an encryption thread, a packet sending thread and the like related to the embodiment of the invention can be replaced by a process or a server and the like, for example, an audio and video data transmission device can call the decryption process to decrypt audio and video data, and the decryption process can comprise one or more decryption threads and the like; for another example, the audio/video data transmission device may invoke the transcoding server to transcode the audio/video data, and the like, and may be flexibly set according to actual needs, and specific content is not limited here.
The various buffers related to the embodiment of the present invention, such as a packet receiving buffer, a buffer to be decrypted, a decrypted buffer, a buffer to be transcoded, a buffer to be encrypted, an encrypted buffer, and a packet sending buffer, may be a ring buffer queue (e.g., a single-read single-write lock-free ring buffer queue), or a shared memory or other storage space, and may be flexibly set according to actual needs, and specific contents are not limited herein. Audio and video data are exchanged between various sub-threads and a main thread through various buffer areas without mutual influence, the data format of the audio and video data can be 'data length + other optional appointed fields + data', and the like, the various sub-threads and the main thread can read the data in the corresponding buffer areas in real time or at preset time intervals, and the specific content is not limited here.
An obtaining unit 302, configured to obtain an encoding format required by each receiving client, where the receiving client is a client of the multiple clients except the sending client.
In the same audio/video session, when there is audio/video data sent by the sending client, the obtaining unit 302 may obtain the coding formats required by the receiving clients except the sending client in the audio/video session. For example, before joining an audio/video session, a client (including a sending client and a receiving client) may send a data transmission capability negotiation request to an audio/video data transmission device through a signaling server, where the data transmission capability negotiation request may carry an audio data format, a video data format, and the like supported by the client, and the audio data format and the video data format are encoding formats of audio/video data. When a data transmission capability negotiation request is received, the audio and video data transmission device can store an audio data format, a video data format and the like supported by a client, determine a coding format (namely a data format) when audio and video data transmission is carried out with the client according to the audio data format and the video data format supported by the client and the audio data format and the video data format supported by the audio and video data transmission device, and send a data transmission capability negotiation response to the client through a signaling server, wherein the data transmission capability negotiation response comprises the coding format and the like when the audio and video data transmission is carried out with the client.
Therefore, the obtaining unit 302 may send an encoding format obtaining request to each receiving client and receive an encoding format returned by each receiving client, or the obtaining unit 302 may obtain an encoding format required by each receiving client from a database of encoding formats required by each client stored in advance, where the encoding formats include an audio encoding format, a video encoding format, and the like.
And the transcoding unit 303 is configured to transcode the audio and video data respectively according to the coding format required by each receiving client, so as to obtain transcoded audio and video data corresponding to each receiving client.
After obtaining the encoding format required by each receiving client, the transcoding unit 303 may determine whether to transcode the audio/video data according to the encoding format of the audio/video data and the encoding format required by each receiving client. The encoding formats of the Audio and video data include an Audio encoding format and a video encoding format, wherein the Audio encoding format may include Advanced Audio Coding (AAC), OPUS, g.711, g.719, and silk, and the like, and the video encoding format may include h264, h265, vp8, vp9, and the like.
Since different types of clients exist in the audio/video session, the encoding formats required by the clients may be different, and at this time, when the encoding format of the audio/video data is different from the encoding format required by each receiving client, it is determined that the audio/video data needs to be transcoded, and the transcoding unit 303 may transcode the audio/video data respectively according to the encoding formats required by the receiving clients to obtain transcoded audio/video data corresponding to the receiving clients. For example, the audio data may be transcoded according to the audio coding format required by each receiving client, so as to obtain transcoded audio data corresponding to each receiving client; and transcoding the video data according to the video coding format required by each receiving client to obtain the transcoded video data corresponding to each receiving client.
It can be understood that when the encoding format required by a certain receiving client in each receiving client is consistent with the encoding format of the audio and video data sent by the sending client, the audio and video data corresponding to the certain receiving client does not need to be transcoded.
For example, when the audio encoding format is g.711 and the video encoding format is h264 in the audio and video data of the sending client, the audio encoding format required by the receiving client a is AAC and the video encoding format required by the receiving client a is h 265; the audio encoding format required by the receiving client B is OPUS and the video encoding format required is vp 9. At this time, the transcoding unit 303 may transcode the audio coding format g.711 into AAC and transcode the video coding format h264 into h265, so as to obtain transcoded audio data AAC and video data h265 corresponding to the receiving client a; and transcoding g.711 in the audio coding format into OPUS, and transcoding h264 in the video coding format into vp9, so as to obtain transcoded audio data OPUS and video data vp9 and the like corresponding to the receiving client B.
In some embodiments, the obtaining unit 302 may specifically be configured to: and acquiring the attribute information of each receiving client, and determining the coding format required by each receiving client according to the attribute information.
The transcoding unit 303 may include a writing subunit, a transcoding subunit, and the like, and specifically may be as follows:
the write-in subunit is used for calling the main thread to read the decrypted audio and video data and writing the decrypted audio and video data into a buffer area to be code converted;
and the transcoding subunit is used for calling an audio and video transcoding thread to read the decrypted audio and video data from the buffer area to be transcoded and transcoding the decrypted audio and video data respectively according to the coding format required by each receiving client to obtain the transcoded audio and video data corresponding to each receiving client.
Specifically, the obtaining unit 302 may obtain attribute information of each receiving client, where the attribute information may include a type of the receiving client, an audio coding format and a video coding format supported by the receiving client, and the like, for example, the obtaining unit 302 may send an attribute information obtaining request to each receiving client, and receive attribute information returned by each receiving client; alternatively, the obtaining unit 302 may store the attribute information of the client in advance when establishing connection with the receiving client, and when obtaining the attribute information is required, may obtain the attribute information of each receiving client from a database storing the attribute information of the client, and the like. The obtaining unit 302 may determine a coding format required by each receiving client according to the attribute information, and at this time, if the audio/video data is decrypted, the writing subunit may invoke the main thread to read the decrypted audio/video data from the decrypted buffer area, and write the decrypted audio/video data into the buffer area to be transcoded. And the transcoding subunit calls an audio and video transcoding thread to read the decrypted audio and video data from the buffer area to be transcoded, and transcodes the decrypted audio and video data respectively according to the coding format required by each receiving client to obtain the transcoded audio and video data corresponding to each receiving client.
It can be understood that, if the audio and video data does not need to be decrypted, the transcoding subunit may directly invoke the audio and video transcoding thread to read the audio and video data from the buffer area to be transcoded, and transcode the audio and video data respectively according to the coding format required by each receiving client, so as to obtain the transcoded audio and video data corresponding to each receiving client.
In some embodiments, the write subunit may be specifically configured to: calling a main thread to read the decrypted audio and video data, and writing the audio data in the decrypted audio and video data into an audio buffer area to be transcoded; and writing the video data in the decrypted audio and video data into a video buffer area to be coded through the main thread.
The transcoding subunit may specifically be configured to:
calling an audio transcoding thread to read audio data from an audio buffer area to be transcoded, and transcoding the audio data according to the audio coding format required by each receiving client to obtain transcoded audio data corresponding to each receiving client;
calling a video transcoding thread to read video data from a video buffer area to be transcoded, and transcoding the video data according to a video coding format required by each receiving client to obtain transcoded video data corresponding to each receiving client;
and setting the transcoded audio data and the transcoded video data corresponding to each receiving client as the transcoded audio/video data corresponding to each receiving client.
The write-in subunit and the transcoding subunit can respectively process audio data and video data in the audio and video data, and specifically, the write-in subunit can call the main thread to read the decrypted audio and video data from the decrypted buffer area, write the audio data in the decrypted audio and video data into an audio buffer area to be transcoded, and write the video data in the decrypted audio and video data into a video buffer area to be transcoded. At this time, the transcoding subunit may call an audio transcoding thread to read audio data from the audio buffer to be transcoded, and transcode the audio data according to the audio coding format required by each receiving client, so as to obtain transcoded audio data corresponding to each receiving client. And the transcoding subunit can call a video transcoding thread to read video data from a video buffer area to be transcoded, and transcode the video data according to the video coding format required by each receiving client, so as to obtain transcoded video data corresponding to each receiving client.
The transcoded audio data corresponding to each receiving client and the transcoded video data corresponding to each receiving client are the transcoded audio/video data corresponding to each receiving client. The audio and video data transmission device can independently realize the heavier functions such as decryption, audio transcoding, video transcoding and the like according to the function and load difference, can flexibly configure the required functions according to the service requirements, and can configure the resource allocation of each functional block, thereby realizing different types of clients such as SDK clients, small program clients, H5 clients, PSTN clients and the like without being limited by the respective audio and video coding and decoding difference and the like, solving the requirements of audio and video data audio coding and decoding formats, video coding and decoding formats and the like required by different clients and realizing the fusion and intercommunication of audio and video data between the different types of clients.
It should be noted that transcoding audio data and video data may share one thread, for example, a transcoding thread, that is, the transcoding subunit may invoke the transcoding thread to transcode audio data, or invoke the transcoding thread to transcode video data. For example, when audio data needs to be transcoded, the transcoding subunit may invoke a transcoding thread to read the audio data from the buffer area to be transcoded, and transcode the audio data; when the video data needs to be transcoded, the transcoding subunit can call a transcoding thread to read the video data from the buffer area to be transcoded and transcode the video data.
The transmission unit 304 is configured to transmit the transcoded audio/video data corresponding to each receiving client.
After the transcoded audio/video data corresponding to each receiving client is obtained, the transmission unit 304 may transmit the transcoded audio/video data corresponding to each receiving client.
In some embodiments, as shown in fig. 9, the audio-video data transmission apparatus may further include: and the encryption unit 306 is configured to encrypt the transcoded audio and video data to obtain encrypted audio and video data when the transcoded audio and video data needs to be encrypted.
The transmission unit 304 may be specifically configured to: and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
Specifically, the encryption unit 306 may determine whether the transcoded audio/video data needs to be encrypted, for example, may determine whether the transcoded audio/video data needs to be encrypted according to the type of the receiving client, and when the receiving client is an SDK client, the audio/video data does not need to be encrypted; when the receiving client is a public client, a browser client or the like, the audio and video data may need to be encrypted. For another example, whether the transcoded audio and video data needs to be encrypted can be judged according to the type of the audio and video data. When it is determined that the transcoded audio/video data needs to be encrypted, the encryption unit 306 may encrypt the transcoded audio/video data to obtain encrypted audio/video data.
In some embodiments, the encryption unit 306 may be specifically configured to:
calling a main thread to read the transcoded audio and video data and acquiring the type of the transcoded audio and video data;
when determining that the transcoded audio and video data needs to be encrypted according to the type of the transcoded audio and video data, writing the transcoded audio and video data into a buffer area to be encrypted;
calling an encryption thread to read transcoded audio and video data from a buffer area to be encrypted, encrypting the transcoded audio and video data according to a preset encryption key, and writing the encrypted audio and video data into an encrypted buffer area;
the transmission unit is further specifically configured to: calling a main thread to read the encrypted audio and video data from the encrypted buffer area and writing the encrypted audio and video data into a packet sending buffer area;
and calling a packet sending thread to read the encrypted audio and video data from the packet sending buffer area, and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
Specifically, the encryption unit 306 may call the main thread to read transcoded audio/video data from the transcoded buffer region, acquire the type of the transcoded audio/video data, and determine whether to encrypt the transcoded audio/video data according to the type of the transcoded audio/video data. For example, the transcoded audio/video data may be analyzed, a packet header is extracted from the transcoded audio/video data, and a type field for storing the type of the audio/video data in the packet header is extracted, and at this time, the type of the audio/video data may be determined according to the content of the type field in the packet header, for example, the audio/video data received by an SDK client, or the audio/video data received by clients such as an applet client, a PSTN client, and a Web client.
When determining that the transcoded audio/video data needs to be encrypted according to the type of the transcoded audio/video data, the encryption unit 306 may write the transcoded audio/video data into a packet sending buffer area; when determining that the transcoded audio/video data needs to be encrypted according to the type of the transcoded audio/video data, the encryption unit 306 may write the transcoded audio/video data into a buffer to be encrypted.
When encryption is needed, the encryption unit 306 may call an encryption thread to read transcoded audio/video data from the buffer to be encrypted, encrypt the transcoded audio/video data according to a preset encryption key, and write the encrypted audio/video data into the encrypted buffer; the preset encryption key may be an encryption key negotiated when the audio/video data transmission device establishes a secure connection with the client.
It should be noted that when the audio and video data are encrypted, the audio data and the video data in the audio and video data may be encrypted respectively, and when a plurality of audio data and a plurality of video data need to be encrypted, the encryption unit 306 may invoke a plurality of encryption threads according to actual needs to encrypt the plurality of audio data and the plurality of video data in parallel.
At this time, the encryption unit 306 may call the main thread to read the encrypted audio/video data from the encrypted buffer, and write the encrypted audio/video data into the packet sending buffer. And then calling a packet sending thread to read the encrypted audio and video data from the packet sending buffer area, and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
The audio and video data transmission device can independently perform heavier functions such as security encryption and decryption, audio transcoding, video transcoding, data transmission and the like according to the function and load difference, can flexibly configure required functions according to business requirements, can configure resource allocation of each functional block, and can efficiently utilize resources; the method and the system realize different types of clients such as the SDK client, the applet client, the PSTN client, the Web client and the like without being limited by respective audio and video coding and decoding differences, safety transmission requirement differences and the like, solve the requirements of audio and video data audio coding and decoding formats, video coding and decoding formats, transmission safety and the like required by different clients and realize the fusion and intercommunication of audio and video data among different types of clients.
It should be noted that, the encryption of the audio and video data and the encryption of the audio and video data may share one thread, for example, an encryption and decryption thread, that is, the encryption unit 306 may invoke the encryption and decryption thread to encrypt the audio and video data, and may invoke the encryption and decryption thread to encrypt the audio and video data.
For example, when the audio/video data needs to be decrypted, the encryption unit 306 may invoke an encryption/decryption thread to read the audio/video data from the buffer to be encrypted/decrypted, and extract a preset field from the audio/video data, where the preset field is used to store an encryption instruction or a decryption instruction, so that the encryption/decryption thread may encrypt the audio/video data according to the encryption instruction, or may decrypt the audio/video data according to the decryption instruction. When it is determined that the decryption operation is required according to the decryption instruction in the preset field, the encryption unit 306 may decrypt the audio and video data through the encryption and decryption thread, and write the decrypted audio and video data into the encryption and decryption buffer, where the decrypted audio and video data may include a field for storing decrypted information, so that after the main thread reads the audio and video data from the encryption and decryption buffer, the field for storing decrypted information may be extracted from the audio and video data, and the audio and video data is determined to be the decrypted audio and video data according to the decrypted information.
When the audio and video data needs to be encrypted, the encryption unit 306 may call an encryption and decryption thread to read the audio and video data from the buffer area to be encrypted and extract a preset field from the audio and video data, when it is determined that the encryption operation needs to be performed according to an encryption instruction in the preset field, the encryption unit 306 may encrypt the audio and video data through the encryption and decryption thread and write the encrypted audio and video data into the encryption and decryption buffer area, where the encrypted audio and video data may include a field for storing encrypted information, so that after the main thread reads the audio and video data from the encryption and decryption buffer area, the field for storing the encrypted information may be extracted from the audio and video data and determine that the audio and video data is the encrypted audio and video data according to the encrypted information.
As can be seen from the above, the first receiving unit 301 in the embodiment of the present invention may receive, in an audio/video session between multiple clients of different types, audio/video data sent by a sending client, where the sending client is any one of the multiple clients; and, the obtaining unit 302 obtains the coding format required by each receiving client, wherein the receiving client is a client other than the sending client in the plurality of clients; then, the transcoding unit 303 may transcode the audio and video data respectively according to the encoding format required by each receiving client, to obtain transcoded audio and video data corresponding to each receiving client; the transcoded audio/video data corresponding to each receiving client is transmitted to each receiving client by the transmission unit 304. According to the scheme, multiple clients of different types can join the same audio and video session, audio and video data are transcoded in the audio and video session according to the coding formats required by the receiving clients, and the transcoded audio and video data are transmitted to the receiving clients respectively, so that the audio and video data interaction of the clients of different types in the same audio and video session is realized, the limitation that the same client can join the same audio and video session to transmit the audio and video data in the prior art is overcome, and the flexibility of audio and video data transmission is greatly improved.
An embodiment of the present invention further provides a server, which may be a transit server, and as shown in fig. 11, it shows a schematic structural diagram of the server according to the embodiment of the present invention, specifically:
the server may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the server architecture shown in FIG. 11 is not meant to be limiting, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:
the processor 401 is a control center of the server, connects various parts of the entire server using various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the server. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.
The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the server, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.
The server further includes a power supply 403 for supplying power to each component, and preferably, the power supply 403 may be logically connected to the processor 401 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system. The power supply 403 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.
The server may also include an input unit 404, the input unit 404 being operable to receive input numeric or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
Although not shown, the server may further include a display unit and the like, which will not be described in detail herein. Specifically, in this embodiment, the processor 401 in the server loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby implementing various functions as follows:
in audio and video sessions among a plurality of clients of different types, receiving audio and video data sent by a sending client, wherein the sending client is any one of the clients; acquiring coding formats required by all receiving clients, wherein the receiving clients are clients except for a sending client in a plurality of clients; transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; and respectively transmitting the transcoded audio and video data corresponding to each receiving client.
Optionally, after the step of receiving and sending audio and video data sent by the client in audio and video sessions among multiple clients of different types, the audio and video transmission method may further include: acquiring the type of audio and video data; and when the audio and video data need to be decrypted according to the type of the audio and video data, decrypting the audio and video data to obtain the decrypted audio and video data.
Optionally, before the step of transmitting the transcoded audio/video data corresponding to each receiving client, the audio/video data transmission method may further include: when the transcoded audio and video data needs to be encrypted, encrypting the transcoded audio and video data to obtain encrypted audio and video data; the step of transmitting the transcoded audio/video data corresponding to each receiving client respectively may include: and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
As can be seen from the above, the embodiment of the present invention can receive the audio and video data sent by the sending client in the audio and video session among the multiple clients of different types, where the sending client is any one of the multiple clients; acquiring a coding format required by each receiving client, wherein the receiving client is a client except for the sending client in the plurality of clients; then transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; and respectively transmitting the transcoded audio and video data corresponding to each receiving client. According to the scheme, multiple clients of different types can join the same audio and video session, audio and video data are transcoded in the audio and video session according to the coding formats required by the receiving clients, and the transcoded audio and video data are transmitted to the receiving clients respectively, so that the audio and video data interaction of the clients of different types in the same audio and video session is realized, the limitation that the same client can join the same audio and video session to transmit the audio and video data in the prior art is overcome, and the flexibility of audio and video data transmission is greatly improved.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed description of the audio and video data transmission method, and are not described herein again.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.
To this end, the embodiment of the present invention provides a storage medium, in which a plurality of instructions are stored, where the instructions can be loaded by a processor to execute the steps in any one of the audio and video data transmission methods provided by the embodiments of the present invention. For example, the instructions may perform the steps of:
in audio and video sessions among a plurality of clients of different types, receiving audio and video data sent by a sending client, wherein the sending client is any one of the clients; acquiring coding formats required by all receiving clients, wherein the receiving clients are clients except for a sending client in a plurality of clients; transcoding the audio and video data respectively according to the coding format required by each receiving client to obtain transcoded audio and video data corresponding to each receiving client; and respectively transmitting the transcoded audio and video data corresponding to each receiving client.
Optionally, after the step of receiving and sending audio and video data sent by the client in audio and video sessions among multiple clients of different types, the audio and video transmission method may further include: acquiring the type of audio and video data; and when the audio and video data need to be decrypted according to the type of the audio and video data, decrypting the audio and video data to obtain the decrypted audio and video data.
Optionally, before the step of transmitting the transcoded audio/video data corresponding to each receiving client, the audio/video data transmission method may further include: when the transcoded audio and video data needs to be encrypted, encrypting the transcoded audio and video data to obtain encrypted audio and video data; the step of transmitting the transcoded audio/video data corresponding to each receiving client respectively may include: and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the storage medium can execute the steps in any audio/video data transmission method provided by the embodiment of the present invention, the beneficial effects that can be achieved by any audio/video data transmission method provided by the embodiment of the present invention can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.
The method, the device and the storage medium for transmitting audio and video data provided by the embodiment of the invention are described in detail, a specific example is applied in the text to explain the principle and the implementation of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (15)

1. An audio/video data transmission method, comprising:
receiving session requests sent by a plurality of clients of different types, wherein the session requests carry session identifiers and client identifiers;
changing account numbers for the clients according to the client identifications;
when the audio and video session corresponding to the session identifier exists, the client is added into the audio and video session, and when the audio and video session corresponding to the session identifier does not exist, the audio and video session is established according to each client after the account is replaced;
in audio and video sessions among a plurality of clients of different types, receiving audio and video data sent by a sending client, wherein the sending client is any one of the clients;
acquiring a coding format required by each receiving client, wherein the receiving client is a client except the sending client in the plurality of clients;
transcoding the audio and video data respectively according to the coding formats required by the receiving clients to obtain transcoded audio and video data corresponding to the receiving clients;
and respectively transmitting the transcoded audio and video data corresponding to each receiving client.
2. The method according to claim 1, wherein after the step of receiving and sending the audio/video data sent by the client in the audio/video session between the plurality of clients of different types, the method further comprises:
acquiring the type of the audio and video data;
and when the audio and video data need to be decrypted is determined according to the type of the audio and video data, decrypting the audio and video data to obtain the decrypted audio and video data.
3. The audio and video data transmission method according to claim 2, wherein the step of acquiring the type of the audio and video data, and when it is determined that the audio and video data needs to be decrypted according to the type of the audio and video data, decrypting the audio and video data to obtain decrypted audio and video data comprises:
calling a main thread to extract a packet header from the audio and video data, and determining the type of the audio and video data according to a type field in the packet header;
when the audio and video data need to be decrypted is determined according to the type of the audio and video data, the audio and video data are written into a buffer area to be decrypted through the main thread;
and calling a decryption thread to read the audio and video data from the buffer area to be decrypted, decrypting the audio and video data according to a preset decryption key, and writing the decrypted audio and video data into the decrypted buffer area.
4. The audio/video data transmission method according to claim 2, wherein the step of obtaining the coding format required by each receiving client, and transcoding the audio/video data according to the coding format required by each receiving client to obtain transcoded audio/video data corresponding to each receiving client comprises:
acquiring attribute information of each receiving client, and determining a coding format required by each receiving client according to the attribute information;
calling a main thread to read the decrypted audio and video data, and writing the decrypted audio and video data into a buffer area to be code-converted;
and calling an audio and video transcoding thread to read the decrypted audio and video data from a buffer area to be transcoded, and transcoding the decrypted audio and video data respectively according to the coding formats required by the receiving clients to obtain the transcoded audio and video data corresponding to the receiving clients.
5. The audio and video data transmission method according to claim 4, wherein the step of calling the main thread to read the decrypted audio and video data and writing the decrypted audio and video data into a buffer area to be transcoded, calling the audio and video transcoding thread to read the decrypted audio and video data from the buffer area to be transcoded, and transcoding the decrypted audio and video data respectively according to the coding format required by each receiving client to obtain the transcoded audio and video data corresponding to each receiving client comprises the steps of:
calling the main thread to read the decrypted audio and video data, and writing the audio data in the decrypted audio and video data into an audio buffer area to be transcoded;
writing the video data in the decrypted audio and video data into a video buffer area to be coded through the main thread;
calling an audio transcoding thread to read the audio data from an audio buffer area to be transcoded, and transcoding the audio data according to the audio coding format required by each receiving client to obtain transcoded audio data corresponding to each receiving client;
calling a video transcoding thread to read the video data from a video buffer area to be transcoded, and transcoding the video data according to the video coding format required by each receiving client to obtain transcoded video data corresponding to each receiving client;
and setting the transcoded audio data and the transcoded video data corresponding to each receiving client as the transcoded audio and video data corresponding to each receiving client.
6. The method for transmitting audio/video data according to claim 1, wherein before the step of transmitting the transcoded audio/video data corresponding to each receiving client, the method further comprises:
when the transcoded audio and video data needs to be encrypted, encrypting the transcoded audio and video data to obtain encrypted audio and video data;
the step of respectively transmitting the transcoded audio and video data corresponding to each receiving client comprises the following steps:
and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
7. The audio and video data transmission method according to claim 5, wherein the step of encrypting the transcoded audio and video data to obtain encrypted audio and video data when the transcoded audio and video data needs to be encrypted, and transmitting the encrypted audio and video data corresponding to each receiving client respectively comprises:
calling a main thread to read the transcoded audio and video data and acquiring the type of the transcoded audio and video data;
when the transcoded audio and video data needs to be encrypted is determined according to the type of the transcoded audio and video data, writing the transcoded audio and video data into a buffer area to be encrypted;
calling an encryption thread to read the transcoded audio and video data from a buffer area to be encrypted, encrypting the transcoded audio and video data according to a preset encryption key, and writing the encrypted audio and video data into an encrypted buffer area;
calling the main thread to read the encrypted audio and video data from the encrypted buffer area, and writing the encrypted audio and video data into a packet sending buffer area;
and calling a packet sending thread to read the encrypted audio and video data from the packet sending buffer area, and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
8. An audio-video data transmission apparatus, comprising:
the first receiving unit is used for receiving audio and video data sent by a sending client in audio and video sessions among a plurality of clients of different types, wherein the sending client is any one of the clients;
an obtaining unit, configured to obtain an encoding format required by each receiving client, where the receiving client is a client of the multiple clients except the sending client;
the transcoding unit is used for transcoding the audio and video data respectively according to the coding formats required by the receiving clients to obtain transcoded audio and video data corresponding to the receiving clients;
the transmission unit is used for respectively transmitting the transcoded audio and video data corresponding to each receiving client;
the second receiving unit is used for receiving session requests sent by a plurality of clients of different types, wherein the session requests carry session identifiers and client identifiers;
the replacing unit is used for replacing the account numbers for the clients according to the client identifications;
and the establishing unit is used for adding the client into the audio and video session when the audio and video session corresponding to the session identifier exists, and establishing the audio and video session according to each client after the account is replaced when the audio and video session corresponding to the session identifier does not exist.
9. The audio-visual data transmission device according to claim 8, characterized in that it further comprises:
and the decryption unit is used for acquiring the type of the audio and video data, and decrypting the audio and video data to obtain decrypted audio and video data when the audio and video data is determined to be required to be decrypted according to the type of the audio and video data.
10. The device for transmitting audio/video data according to claim 9, wherein the decryption unit is specifically configured to:
calling a main thread to extract a packet header from the audio and video data, and determining the type of the audio and video data according to a type field in the packet header;
when the audio and video data need to be decrypted is determined according to the type of the audio and video data, the audio and video data are written into a buffer area to be decrypted through the main thread;
and calling a decryption thread to read the audio and video data from the buffer area to be decrypted, decrypting the audio and video data according to a preset decryption key, and writing the decrypted audio and video data into the decrypted buffer area.
11. The device for transmitting audio/video data according to claim 9, wherein the obtaining unit is specifically configured to: acquiring attribute information of each receiving client, and determining a coding format required by each receiving client according to the attribute information;
the transcoding unit comprises:
the write-in subunit is used for calling the main thread to read the decrypted audio and video data and writing the decrypted audio and video data into a buffer area to be code converted;
and the transcoding subunit is used for calling an audio and video transcoding thread to read the decrypted audio and video data from a buffer area to be transcoded and transcoding the decrypted audio and video data respectively according to the coding format required by each receiving client to obtain the transcoded audio and video data corresponding to each receiving client.
12. The device for transmitting audio/video data according to claim 11, wherein the write subunit is specifically configured to:
calling a main thread to read the decrypted audio and video data, and writing the audio data in the decrypted audio and video data into an audio buffer area to be transcoded;
writing the video data in the decrypted audio and video data into a video buffer area to be coded through the main thread;
the transcoding subunit is specifically configured to:
calling an audio transcoding thread to read the audio data from an audio buffer area to be transcoded, and transcoding the audio data according to the audio coding format required by each receiving client to obtain transcoded audio data corresponding to each receiving client;
calling a video transcoding thread to read the video data from a video buffer area to be transcoded, and transcoding the video data according to the video coding format required by each receiving client to obtain transcoded video data corresponding to each receiving client;
and setting the transcoded audio data and the transcoded video data corresponding to each receiving client as the transcoded audio and video data corresponding to each receiving client.
13. The audio-visual data transmission device according to claim 8, characterized in that it further comprises:
the encryption unit is used for encrypting the transcoded audio and video data to obtain the encrypted audio and video data when the transcoded audio and video data needs to be encrypted;
the transmission unit is specifically configured to: and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
14. The device for transmitting audio/video data according to claim 13, wherein the encryption unit is specifically configured to:
calling a main thread to read the transcoded audio and video data and acquiring the type of the transcoded audio and video data;
when the transcoded audio and video data needs to be encrypted is determined according to the type of the transcoded audio and video data, writing the transcoded audio and video data into a buffer area to be encrypted;
calling an encryption thread to read the transcoded audio and video data from a buffer area to be encrypted, encrypting the transcoded audio and video data according to a preset encryption key, and writing the encrypted audio and video data into an encrypted buffer area;
the transmission unit is further specifically configured to: calling the main thread to read the encrypted audio and video data from the encrypted buffer area, and writing the encrypted audio and video data into a packet sending buffer area;
and calling a packet sending thread to read the encrypted audio and video data from the packet sending buffer area, and respectively transmitting the encrypted audio and video data corresponding to each receiving client.
15. A storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps of the method for audio-visual data transmission according to any one of claims 1 to 7.
CN201810333639.XA 2018-04-13 2018-04-13 Audio and video data transmission method and device and storage medium Active CN110213652B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810333639.XA CN110213652B (en) 2018-04-13 2018-04-13 Audio and video data transmission method and device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810333639.XA CN110213652B (en) 2018-04-13 2018-04-13 Audio and video data transmission method and device and storage medium

Publications (2)

Publication Number Publication Date
CN110213652A CN110213652A (en) 2019-09-06
CN110213652B true CN110213652B (en) 2021-11-30

Family

ID=67779051

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810333639.XA Active CN110213652B (en) 2018-04-13 2018-04-13 Audio and video data transmission method and device and storage medium

Country Status (1)

Country Link
CN (1) CN110213652B (en)

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110740351B (en) * 2019-11-14 2022-05-24 普联技术有限公司 Method, device and storage medium for transmitting stream media data
CN111107393A (en) * 2019-11-29 2020-05-05 视联动力信息技术股份有限公司 Video transcoding method, transcoder, electronic device and storage medium
CN111064916A (en) * 2019-12-02 2020-04-24 视联动力信息技术股份有限公司 Video call method and device, electronic equipment and storage medium
CN111131743A (en) * 2019-12-25 2020-05-08 视联动力信息技术股份有限公司 Video call method and device based on browser, electronic equipment and storage medium
CN111294551B (en) * 2020-02-25 2021-06-08 广州华多网络科技有限公司 Method, device and equipment for audio and video transmission and storage medium
CN111479095B (en) * 2020-06-24 2020-10-02 腾讯科技（深圳）有限公司 Service processing control system, method and device
CN112333475A (en) 2020-10-29 2021-02-05 京东方科技集团股份有限公司 Processing method and processing system of streaming media data and storage server
CN113515235A (en) * 2021-07-12 2021-10-19 深圳市明源云客电子商务有限公司 Voice recording method, device, charging dock and storage medium
CN115914677A (en) * 2022-09-19 2023-04-04 上海辰锐信息科技有限公司 Intelligent video safety networking device and server
CN117692598A (en) * 2024-02-04 2024-03-12 浙江华创视讯科技有限公司 Video stream sending method and device

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN201491019U (en) * 2009-08-25 2010-05-26 北京原力创新科技有限公司 Network video system supporting multiple terminals
JP2012199977A (en) * 2012-05-25 2012-10-18 Nippon Telegr & Teleph Corp <Ntt> Video transmission program
CN102984546A (en) * 2012-11-01 2013-03-20 上海文广互动电视有限公司 Transcoding service system for distributed video transcoding

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN201491019U (en) * 2009-08-25 2010-05-26 北京原力创新科技有限公司 Network video system supporting multiple terminals
JP2012199977A (en) * 2012-05-25 2012-10-18 Nippon Telegr & Teleph Corp <Ntt> Video transmission program
CN102984546A (en) * 2012-11-01 2013-03-20 上海文广互动电视有限公司 Transcoding service system for distributed video transcoding

Also Published As

Publication number Publication date
CN110213652A (en) 2019-09-06

Similar Documents

Publication Publication Date Title
CN110213652B (en) Audio and video data transmission method and device and storage medium
US10778731B2 (en) Communications methods, apparatus and systems for conserving media resource function resources
EP2779579B1 (en) Method and apparatuses for realizing voip call in cloud computing environment
US8380868B2 (en) Method and system for data transmission
US20200220936A1 (en) Methods and apparatus for storing and/or retrieving session state information
US20130282820A1 (en) Method and System for an Optimized Multimedia Communications System
US20170019443A1 (en) Device and method for transferring the rendering of multimedia content
US9736194B1 (en) System for establishing communication between devices
US10523730B2 (en) Real-time transport protocol (RTP) media conference server routing engine
JP2024520245A (en) VoLTE voice encryption communication method, terminal and system
CN109688416B (en) Network live broadcast room access method and network live broadcast system
CN106559396B (en) Media multicast method and system based on Web real-time communication
CN105554029A (en) Method for realizing media intercommunication between WebRTC terminal and SIP terminal and media gateway
WO2016177121A1 (en) File transmission method and device, and application server
CN112422583A (en) Method and system for fusion and intercommunication of multi-protocol video application
CN110875914A (en) Method and device for transmitting messages based on shared session link
CN110798651A (en) Video conference method, device, equipment and storage medium
US9264662B2 (en) Chat preauthorization
US20080025306A1 (en) Internet protocol television system, method for providing internet protocol multicast TV signal, TV transferring apparatus, and TV receiving apparatus
CN117097702A (en) High concurrency WebRTC gateway processing method based on SIP protocol, gateway system, electronic device and storage medium
CN105187678B (en) A kind of method and VoIP server of telephone conference room bridge joint
US9374391B2 (en) Composite endpoint mechanism
US11044214B2 (en) Multimedia file adaption across different communication platforms
CN109769100A (en) Video conference test method, device and storage medium
WO2009030171A1 (en) Media service implementing method and communication system and associated devices

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant