CN113132376B

CN113132376B - Media data processing method, device and system, electronic equipment and storage medium

Info

Publication number: CN113132376B
Application number: CN202110401450.1A
Authority: CN
Inventors: 黄从韬; 金亮; 黄钰良
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-04-14
Filing date: 2021-04-14
Publication date: 2022-11-22
Anticipated expiration: 2041-04-14
Also published as: CN113132376A

Abstract

The embodiment of the disclosure provides a media data processing method, a device, a system, an electronic device and a storage medium. The method is executed by a streaming media server, the streaming media server comprises a media transmission channel and a proxy transmission channel, and the method comprises the following steps: receiving media data to be issued which is packaged by a webpage communication media protocol from a target page browsing terminal through a media transmission channel; sending the media data to be issued, which is encapsulated by adopting a webpage communication media protocol, to an agent transmission channel through a media transmission channel; converting the media data to be published packaged by the webpage communication media protocol into the media data to be published packaged by the client media protocol by using the proxy transmission channel; and sending the media data to be issued which is encapsulated by the client media protocol to a target client through the proxy transmission channel.

Description

Media data processing method, device and system, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer and communication technologies, and in particular, to a method, an apparatus, a system, an electronic device, and a computer-readable storage medium for processing media data.

Background

In the related art, since media protocols used by a Web (Web) browsing page side and a client side are different, smooth media communication cannot be performed between a user using the browsing page (which may be referred to as a Web user) and a user using the client side (which may be referred to as a client user).

Disclosure of Invention

The embodiment of the disclosure provides a media data processing method, a device, a system, an electronic device and a computer-readable storage medium, which can realize transmission of media data between a target page browsing terminal and a target client terminal adopting different media protocols.

The embodiment of the disclosure provides a media data processing method, which is executed by a streaming media server, wherein the streaming media server comprises a media transmission channel and an agent transmission channel. The method comprises the following steps: receiving media data to be issued which is packaged by a webpage communication media protocol from a target page browsing terminal through a media transmission channel; sending the media data to be issued, which is encapsulated by a webpage communication media protocol, to an agent transmission channel through a media transmission channel; converting the media data to be published packaged by the webpage communication media protocol into the media data to be published packaged by the client media protocol by using the proxy transmission channel; and sending the media data to be issued, which is encapsulated by the client media protocol, to the target client through the proxy transmission channel.

The disclosed embodiment provides a media data processing device, which is arranged on a streaming media server, wherein the streaming media server comprises a media transmission channel and an agent transmission channel. The device comprises: the media data receiving unit to be issued is used for receiving the media data to be issued which is encapsulated by adopting a webpage communication media protocol from the target page browsing terminal through a media transmission channel; the media data forwarding unit to be issued is used for sending the media data to be issued, which is encapsulated by adopting a webpage communication media protocol, to the proxy transmission channel through the media transmission channel; the media data conversion unit to be issued is used for converting the media data to be issued, which is encapsulated by the webpage communication media protocol, into the media data to be issued, which is encapsulated by the client media protocol, by using the proxy transmission channel; and the to-be-published media data sending unit is used for sending the to-be-published media data packaged by the client media protocol to the target client through the proxy transmission channel.

The embodiment of the disclosure provides a media data transmission system, which comprises a streaming media server, wherein the streaming media server comprises a media transmission channel and an agent transmission channel. The media transmission channel is used for receiving the to-be-published media data packaged by the webpage communication media protocol from the target webpage login terminal and sending the to-be-published media data packaged by the webpage communication media protocol to the proxy transmission channel; the proxy transmission channel is used for converting the to-be-published media data encapsulated by the webpage communication media protocol into the to-be-published media data encapsulated by the client media protocol and sending the to-be-published media data encapsulated by the client media protocol to the target client login terminal.

The embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the media data processing method as in the above embodiments.

An embodiment of the present disclosure provides an electronic device, including: at least one processor; a storage device configured to store at least one program that, when executed by at least one processor, causes the at least one processor to implement the media data processing method as in the above embodiments.

According to an aspect of the present disclosure, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the media data processing method provided in the various alternative implementations of the above-mentioned embodiments.

In the technical solutions provided by some embodiments of the present disclosure, an agent transmission channel is added in a streaming media server, and the agent transmission channel has a function of converting between a web communication media protocol and a client media protocol, so that when a media transmission channel in the streaming media server receives media data to be distributed encapsulated by the web communication media protocol from a target web browser, the media data to be distributed encapsulated by the web communication media protocol can be forwarded to the agent transmission channel, and the agent transmission channel can convert the media data to be distributed encapsulated by the web communication media protocol into media data to be distributed encapsulated by the client media protocol, and transmit the media data to be distributed encapsulated by the client media protocol to a corresponding target client by means of the agent transmission channel, thereby implementing intercommunication between the target web browser and the target client based on the web communication media protocol, that is, implementing interconnection between the target web browser and the target client using different media protocols.

Drawings

Fig. 1 shows a schematic diagram of a streaming server in the related art.

Fig. 2 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure.

Fig. 3 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure.

Fig. 4 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure.

Fig. 5 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure.

Fig. 6 schematically shows a schematic diagram of a streaming media server according to an embodiment of the present disclosure.

Fig. 7 schematically shows a schematic diagram of a media data transmission system according to an embodiment of the present disclosure.

Fig. 8 schematically shows a transmission timing diagram of an upstream media data stream according to an embodiment of the present disclosure.

Fig. 9 schematically shows a transmission timing diagram of a downstream media data stream according to an embodiment of the present disclosure.

Fig. 10 schematically shows a block diagram of a media data processing device according to an embodiment of the present disclosure.

FIG. 11 shows a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.

The described features, structures, or characteristics of the disclosure may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and steps, nor do they necessarily have to be performed in the order described. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element/component/etc.; the terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements/components/etc. other than the listed elements/components/etc.; the terms "first," "second," and "third," etc. are used merely as labels, and are not limiting on the number of their objects.

WebRTC (Web Real-Time Communication ) is one of important characteristics supported by HTML5 (Hyper Text Markup Language 5, hypertext Markup Language 5, abbreviated as H5), and is a technology capable of supporting a Web browser to perform Real-Time multimedia Communication, unlike a traditional multimedia Communication method based on a local client or a browser plug-in, webRTC integrates core modules, such as processing (acquisition, encoding, and enhancement), network transmission, session control, and the like, necessary for multimedia Communication into the browser, so that a third-party Application developer can obtain a Real-Time multimedia Communication capability only by calling a simple JavaScript (JS) (a lightweight, interpreted, or instant-compilation Programming Language with function priority, abbreviated as JS) API (Application Programming Interface).

Fig. 1 shows a schematic diagram of a streaming server in the related art.

As shown in fig. 1, taking the open source Janus streaming server as an example, the server is mainly composed of three parts, namely a Core (Core) 101, a plug-in (plug) 102, and a Transport channel (Transport) 103.

The Core 101 is used for processing forwarding of data streams, accessing various protocols, including ICE (Interactive Connectivity Establishment), DTLS (Datagram Transport Layer Security), RTP (Real-time Transport Protocol), RTCP (Real-time Transport Control Protocol), SRTP (Secure Real-time Transport Protocol), SCTP (Stream Control Protocol ), a Protocol for transmitting multiple data streams simultaneously between two ends of a network connection, a service provided by the sip and UDP (User Datagram Protocol) and TCP (Transmission Control Protocol), a Session Control Protocol (Session Control Protocol), a Session Description Protocol (Session Description Protocol), a Session Description Protocol (Session transfer Protocol), a Session Description Protocol (Session Control Protocol), a Session transfer Protocol (Session transfer Protocol), a Session transfer Protocol (Session Description Protocol), a Session Description Protocol (Session transfer Protocol), and a media Protocol (Session Description Protocol), and the like.

The service management of Janus is managed in the way of Plugin 102, so that developers can realize own service plug-ins in Janus according to own needs. In fact, for general requirements, janus already has related plug-ins, such as video room (VideoRoom), video call (VideoCall), text room (TextRoom), streaming (Streaming), SIP (Session Initiation Protocol), audio room (AudioRoom). The video room plug-in can be used for multi-person audio and video interaction.

Transport 103 is the signaling Transport layer for Janus. Janus does not limit the signaling Protocol used by the signaling interface, and currently supported protocols include HTTP (HyperText Transfer Protocol), webSocket (web socket), MQTT (Message queue Telemetry Transport), nanoMsg (a Message queue), and RabbitMQ (a Message queue).

However, the streaming media server shown in fig. 1 can only implement interconnection and interworking between multiple Web users (for example, users logged in through a Web browser) and users based on the WebRTC transmission protocol stack, but cannot implement interconnection and interworking between the Web users and users in a client media protocol, for example, a private audio/video protocol system. The client media protocol, such as a private audio/video protocol, is different from an open-source transmission protocol stack, such as WebRTC, so that more refined network transmission control can be achieved, and the client media protocol, such as the private audio/video protocol, is difficult to replace at present.

Based on the above description, the embodiments of the present disclosure provide a media data processing method. The media data processing method provided by each embodiment of the present disclosure may be executed by any of the above mentioned electronic devices, and the electronic device may be a server (including a streaming media server, a signaling server, a media server, and the like), may also be a terminal (including a target web page login terminal and a target client login terminal, and the like), and may also be implemented by interaction between the server and the terminal.

The server in the embodiment of the present disclosure may be an independent server, a server cluster or a distributed system formed by a plurality of servers, or a cloud server providing a cloud computing service. The terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein.

The following detailed description of exemplary embodiments of the disclosure refers to the accompanying drawings.

Fig. 2 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure. The method provided by the embodiment of fig. 2 is exemplified by being executed by a streaming media server, but the disclosure is not limited thereto. The method provided by the embodiment of the disclosure can be applied to application scenes related to media data transmission, such as video on demand, video conference, video chat, voice chat, remote education, telemedicine, online live broadcast and the like.

In the embodiments of the present disclosure, streaming media refers to a media form that transmits media data such as audio, video, and multimedia files in a network in a streaming manner. Compared with the network playing mode of downloading and watching, the streaming media can compress continuous audio and video information and the like and then put the compressed information on a network server, and a user can watch the information while downloading the information without waiting for the completion of the downloading of the whole file.

In the embodiment of the present disclosure, the main functions of the streaming Media Server (Media Server) are to collect, cache, schedule, and transmit and play streaming Media content, for example, audio and video data streams of a Web user can be processed and forwarded to other corresponding users. The streaming media server may be any one or more streaming media servers selected to process the currently pending media data. In the following description, the streaming media server is exemplified as mediasound, but the disclosure is not limited thereto.

The mediasound is a streaming media server in an SFU (Selective Forwarding Unit) mode with an open source.

As shown in fig. 2, the method provided by the embodiment of the present disclosure may include the following steps.

In step S210, the media data to be published, which is encapsulated by the web communication media protocol, is received from the target web browser through the media transmission channel.

In the embodiment of the disclosure, the web page communication media protocol refers to a protocol capable of realizing media communication at a web browsing end. In the following description, the web communication media protocol is a WebRTC transmission protocol stack (abbreviated as a web real-time communication protocol), but the disclosure is not limited thereto. Correspondingly, the media transmission channel based on the web communication media protocol may be labeled as WebRTCTransport, that is, a transmission channel based on the WebRTC transmission protocol stack or the web real-time communication protocol.

In the embodiment of the present disclosure, the transmission channel (Transport) in the streaming media server may include the above media transmission channel based on the web communication media protocol, and may further include a following proxy transmission channel, where the transmission channel may be based on UDP or SCTP, and the like.

In an exemplary embodiment, the media transmission channel may include a media transmission channel for publishing a stream (Publish), and may be used to implement transmission of uplink media data (i.e., media data to be published) from the target page view end to the target client.

In the embodiment of the present disclosure, the target page browsing end refers to an end of a user (taking user a as an example) browsing a Web page, for example, a certain browser installed on the target page browsing end browses the target Web page, and a terminal corresponding to the target page browsing end may be referred to as a target Web page login terminal (taking terminal a as an example). In the following description, the user a initiates transmission of real-time media data, such as real-time voice, real-time video, and the like, through a WebRTC module in a browser on a target Web page login terminal.

In the embodiment of the present disclosure, the media data to be published refers to uplink media data, such as an uplink voice stream, an uplink video stream, and the like, which is currently prepared to be sent to a target client by a Web user or a browser user.

In step S220, the media data to be published, which is encapsulated by the web communication media protocol, is sent to the proxy transmission channel through the media transmission channel.

In the embodiment of the present disclosure, the proxy transmission channel has a function of interconversion between a web communication media protocol and a client media protocol in the following, and has a function of forwarding media data.

In an exemplary embodiment, the proxy transport channel may include a proxy transport channel for a subscription stream (Subscribe).

In an exemplary embodiment, before receiving, from the target page browser terminal through the media transmission channel, the media data to be published encapsulated by the web communication media protocol, the method may further include: receiving a first producer role creation request message initiated by a target page browsing terminal through a media transmission channel for issuing a stream; responding to the first producer role creation request message by the media transmission channel for publishing the stream, generating a first producer role on the media transmission channel for publishing the stream, and storing a first producer role identifier of the first producer role; sending a first consumer role creation request message to a proxy transmission channel for subscribing to the stream through a media transmission channel for publishing the stream; receiving and responding to the first consumer role creation request message through a proxy transmission channel for the subscription stream, and generating a first consumer role on the proxy transmission channel for the subscription stream; and the proxy transmission channel for subscribing the stream enables the first consumer role to subscribe the first producer role identification, and forms a publishing and subscribing relationship between the first consumer role and the first producer role. Further, a first consumer role identification for the first consumer role can also be saved.

In an exemplary embodiment, before receiving the first producer role creation request message initiated by the target page view terminal through the media transmission channel for publishing the stream, the method may further include: receiving a first transmission channel creation request message initiated by a target page browsing terminal through a media stream processing process in a streaming media server; the media stream processing process responds to the first transmission channel creation request message and creates a media transmission channel for publishing the stream and a proxy transmission channel for subscribing the stream in the media stream processing process; and the media stream processing process generates an Identifier (ID) of a media transmission channel for publishing the stream and an identifier of a proxy transmission channel for subscribing the stream, and returns the identifiers of the media transmission channel for publishing the stream and the proxy transmission channel for subscribing the stream to the target page browsing terminal.

In step S230, the to-be-published media data encapsulated by the web communication media protocol is converted into to-be-published media data encapsulated by the client media protocol by using the proxy transmission channel.

In the embodiment of the present disclosure, the client media protocol refers to a protocol for implementing client media data communication, for example, a private audio/video protocol adopted by a client. The private audio/video protocol refers to an audio/video protocol with unpublished protocol content and can be customized according to actual requirements. In the following example, if the client media protocol is a private audio protocol in a private audio/video protocol, the proxy transport channel may be marked as AudioTransport.

It is to be understood that, when the media data (including the above-mentioned media data to be published and the following media data to be subscribed) is audio and video data, the private audio and video protocol in the embodiment of the present disclosure may only include a private audio protocol (e.g. a voice session scene), may only include a private video protocol, or include both the private audio protocol and the private video protocol, which is not limited by the present disclosure.

In the embodiment of the disclosure, the proxy transmission channel knows the encoding and decoding rules of the web communication media protocol, such as WebRTC, and the client media protocol, so that when the proxy transmission channel receives the to-be-published media data encapsulated by the web communication media protocol, the to-be-published media data can be analyzed from the to-be-published media data by using the decoding rules of the web communication media protocol, and then the to-be-published media data is repackaged by using the encoding rules of the client media protocol to generate the to-be-published media data encapsulated by the client media protocol.

In step S240, the media data to be published, which is encapsulated by the client media protocol, is sent to the target client through the proxy transmission channel.

In the embodiment of the present disclosure, a target client refers to a local client installed on a terminal by another user (i.e., a user other than the browser user), and a corresponding terminal is referred to as a target client login terminal, for example, if the terminal is a mobile terminal, the client installed on the terminal is referred to as a mobile APP (Application) terminal, and the corresponding user is referred to as a mobile APP user; if the terminal is a non-mobile terminal, for example, a PC terminal, the client installed on the terminal is called a PC client, and the corresponding user is called a PC client user.

In the embodiment of the present disclosure, the target client performs media data communication with the corresponding media server by using the client media protocol, and the media server may process the to-be-published media data using the client media protocol, for example, determine which target clients to send, and transmit the to-be-published media data using the client media protocol to the one or more target clients (i.e., one-to-one communication or one-to-many communication). In the step S230, the proxy transmission channel in the streaming media server has converted the to-be-published media data sent by the browser user from the web communication media protocol to the client media protocol corresponding to the target client, so that the to-be-published media data can be transmitted to the target client through the proxy transmission channel.

The client in the embodiment of the present disclosure may be any one of a game client, a social client, and the like. The social client is software for realizing information interaction between users through a network. The social client may include at least one of: instant messaging clients, and the like. Instant messaging refers to a terminal service that allows two or more people to use a network to transmit information such as text, files, voice, etc. and to communicate with audio and video instantly.

In an exemplary embodiment, receiving, from the target page browser end through the media transmission channel, the to-be-published media data encapsulated by using a web communication media protocol may include: and receiving the media data to be published packaged by adopting a webpage communication media protocol from a target page browser terminal through a media transmission channel for publishing the stream.

The sending, to the proxy transmission channel, the to-be-published media data encapsulated by using the web communication media protocol through the media transmission channel may include: based on the publishing and subscribing relationship between the first consumer role and the first producer role, sending the to-be-published media data packaged by adopting a webpage communication media protocol to a proxy transmission channel for subscribing the stream through a media transmission channel for publishing the stream.

The converting, by using the proxy transmission channel, the to-be-published media data encapsulated by the web communication media protocol into the to-be-published media data encapsulated by the client media protocol may include: analyzing media data to be published packaged by adopting a webpage communication media protocol by utilizing an agent transmission channel for subscribing the stream; and the proxy transmission channel for subscribing the stream converts the media data to be published obtained by analysis into the media data to be published packaged by adopting a client media protocol.

The method for sending the media data to be issued, which is encapsulated by the client media protocol, to the target client through the proxy transmission channel comprises the following steps: and sending the media data to be published which is encapsulated by the client media protocol to a target client through a proxy transmission channel for subscribing the stream.

The media data processing method provided by the embodiment of the disclosure adds an agent transmission channel in a streaming media server, and enables the agent transmission channel to have a function of converting between a web page communication media protocol and a client media protocol, so that when the media transmission channel in the streaming media server receives to-be-published media data encapsulated by the web page communication media protocol from a target web browser, the to-be-published media data encapsulated by the web page communication media protocol can be forwarded to the agent transmission channel, the agent transmission channel can convert the to-be-published media data encapsulated by the web page communication media protocol into to-be-published media data encapsulated by the client media protocol, and then transmit the to-be-published media data encapsulated by the client media protocol to a corresponding target client by means of the agent transmission channel, thereby realizing the transmission of the to-be-published media data encapsulated by the web page communication media protocol and transmitted by the target web browser to the target client based on the client media protocol, that is interconnected between the target web browser using different media protocols and the target client.

Fig. 3 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure. In the embodiment of fig. 3, a process of transmitting uplink media data (i.e., to-be-published media data) from a target page browsing end to a target client through cooperation between a media transmission channel for publishing a stream and a proxy transmission channel for subscribing to a stream is described. In the embodiment of fig. 3, the web page communication media protocol is a web page real-time communication protocol, and the client media protocol is a private audio/video protocol.

As shown in fig. 3, the method provided by the embodiment of the present disclosure may include the following steps.

In step S301, a first transmission channel creation request message initiated by the target page browsing terminal is received through a media stream processing process in the streaming server.

In the embodiment of the present disclosure, a media stream processing procedure (Worker) in the streaming media server refers to a procedure or a sub-procedure in the streaming media server, which is responsible for processing a media stream, for example, a procedure in the mediasound which is responsible for processing a media stream.

In step S302, the media stream processing procedure responds to the first transmission channel creation request message, and creates a media transmission channel for publishing streams and a proxy transmission channel for subscribing to streams in the media stream processing procedure.

In step S303, the media stream processing process generates an identifier of a media transmission channel for publishing the stream and an identifier of a proxy transmission channel for subscribing to the stream, and returns them to the target page browsing terminal.

In step S304, a first producer role creation request message initiated by the target page view is received through the media transmission channel for publishing the stream.

In an exemplary embodiment, the streaming server may further include a scripting language execution environment, and the scripting language execution environment may be configured to receive the first producer role creation request message from the distributed message queue and send the first producer role creation request message to the media transmission channel for publishing the stream.

In the embodiment of the disclosure, the scripting language operating environment is represented as nodjs, and is a JavaScript operating environment based on a Chrome V8 engine. Taking the streaming media server as the example of the mediasound, the script language operating environment is expressed as mediasound NodeJS. The media sound can include media sound node js and media sound marker, and the media sound node js and the media sound marker can communicate with each other through processes.

In the embodiments of the present disclosure, any suitable distributed message queue may be used as long as it can meet the performance requirements, such as low latency, throughput, reliability, and the like. In the following description, NATS is used as an example for illustration, but the present disclosure is not limited thereto, and in other embodiments, any one of ActiveMQ, kafKa, rabbitMQ, redis, kestrel, NSQ, and the like may be used.

The NATS is an open-source, lightweight and high-performance distributed message queue, supports various message delivery models and comprises the following steps: publish Subscribe (Publish Subscribe), which is a message sent by a publisher on a Subject and received by all subscribers listening to the Subject.

Wherein the distributed message queue may be configured to receive a first producer role creation request message from a user agent in a signaling server.

In the embodiment of the present disclosure, the signaling server (SigServer) may be configured to directly interact with a target page browser corresponding to a Web user through a WebSocket, and receive an instruction/signaling of the Web user, for example, control behaviors such as starting, pausing, and ending of a media stream.

The user agent is essentially a class object, and may contain user information of a target web page on a target page browser, an identifier of a media Transport channel for publishing a stream, an identifier of a proxy Transport channel for subscribing to a stream, and the like, and may further include an identifier of a media Transport channel for subscribing to a stream, an identifier of a proxy Transport channel for publishing a stream, and the like, which may be collectively labeled as a Transport ID, for example, a WebRTCTransport ID that may include a publishing stream and a subscribing stream, an ID of a proxy Transport channel for publishing a stream and subscribing to a stream, and the like.

For example, the user information may contain some basic information of the browser user who logs in the target web page through the target page browser side, such as a user ID, a user login ticket, roomID ID, webSocket connected to WebSDK, and the like. Wherein roomID refers to the identity of the room in which the user is to initiate a communication of voice, video, or the like.

The user agent may be understood as that a browser user creates an object for identifying himself on the signaling server through a browser on the target page browsing side, and the subsequent operation of the browser user through the browser on the target page browsing side is all represented as operating the object to issue a corresponding command to a streaming media server, such as mediasound. The user agent is added in the signaling server and can be used for identifying the browser user, carrying out validity check on the browser user, responding to the operation of the publishing flow and the subscription flow of the browser user, and obtaining a corresponding return value to the browser on the target page browsing end.

In an exemplary embodiment, the user agent may be configured to generate a first producer role creation request message in response to the publish stream signaling sent by the target page browsing terminal, and send the first producer role creation request message to the distributed message queue.

In an exemplary embodiment, the signaling server may be configured to create a user agent in the signaling server in response to the join signaling sent by the target page view end, and control the user agent to send the first transmission channel creation request message to the distributed message queue. The distributed message queue may also be used to send a first transport channel creation request message to the scripting language execution environment. The scripting language runtime environment may also be used to send a first transport channel creation request message to the media stream processing process.

In step S305, the media transmission channel for publishing the stream generates a first producer role on the media transmission channel for publishing the stream in response to the first producer role creation request message and saves a first producer role identification of the first producer role.

Producer refers to a Producer role corresponding to an abstract entity that produces media data (including the above-mentioned media data to be published and the following media data to be subscribed) such as audio or video in the embodiments of the present disclosure. The producer roles in the embodiment of the present disclosure include a first producer role here and a following second producer role, where the first producer role corresponds to the media data to be published generated by the target page browsing end, and the second producer role corresponds to the media data to be subscribed generated by the following target client. A published stream (Publish) corresponds to the Producer's behavior of producing media data, such as video or audio.

In step S306, a first consumer role creation request message is sent to a broker transmission channel for a subscription stream through a media transmission channel for a publish stream.

In step S307, a first consumer role is generated on the proxy transmission channel for the subscription stream by receiving and responding to the first consumer role creation request message through the proxy transmission channel for the subscription stream. Further, a first consumer role identification for the first consumer role may also be saved.

Consumer refers to a Consumer role, corresponding to an abstract entity that consumes media data, such as audio or video, in embodiments of the disclosure. The consumer roles in the embodiment of the present disclosure include a first consumer role here and a second consumer role below, where the first consumer role corresponds to consuming media data to be published, and the second consumer role corresponds to consuming media data to be subscribed. A subscription stream (Subscribe) corresponds to the behavior of the Consumer to receive media data, such as audio or video.

In step S308, the proxy transmission channel for subscribing to the stream enables the first consumer role to subscribe to the first producer role identifier, and forms a publish-subscribe relationship between the first consumer role and the first producer role.

In step S309, the media data to be published, which is encapsulated by the web real-time communication protocol, is received from the target page browser via the media transmission channel for publishing the stream.

In step S310, based on the publish-subscribe relationship between the first consumer role and the first producer role, the media data to be published, which is encapsulated by using the web real-time communication protocol, is sent to the proxy transmission channel for subscribing the stream through the media transmission channel for publishing the stream.

In step S311, the to-be-published media data encapsulated by the web real-time communication protocol is analyzed by using the proxy transmission channel for subscribing the stream, and is converted into to-be-published media data encapsulated by the private audio/video protocol.

In an exemplary embodiment, a media service object may be included within a proxy transport channel for a subscription stream, and a user datagram protocol data channel is maintained between the media service object and a media server.

The parsing, by using the proxy transmission channel for subscribing the stream, the media data to be published encapsulated by the web communication media protocol, and converting the media data to be published encapsulated by the client media protocol, may include: and analyzing the to-be-published media data encapsulated by adopting the webpage communication media protocol by utilizing the media service object, and converting the to-be-published media data encapsulated by adopting the client media protocol.

In step S312, the to-be-published media data encapsulated by the private audio/video protocol is sent to the media server adopting the private audio/video protocol through the proxy transmission channel for subscribing the stream.

The sending, through the proxy transmission channel, the to-be-published media data encapsulated by the client media protocol to the media server using the client media protocol may include: and sending the media data to be issued which is encapsulated by the media protocol of the client to a media server by utilizing a user datagram protocol data channel (UDP data channel).

In step S313, the media server sends the to-be-published media data encapsulated by the private audio/video protocol to the target client.

The media data processing method provided by the embodiment of the present disclosure can implement transmission of uplink media data from a target page browsing end to a target client through a media transmission channel for publishing a stream and a proxy transmission channel for subscribing a stream in a streaming media server, that is, implement transmission of to-be-published media data, which is sent by the target page browsing end and encapsulated based on a Web communication media protocol, to the target client based on the client media protocol, that is, to-be-published media data, such as audio and video data, generated by a user Web browser can be injected into a background server or a media server based on the client media protocol, such as a private audio and video protocol, through the proxy transmission channel, thereby solving the problem of uplink media data exchange between the Web communication media protocol and the client media protocol. The method and the system realize the transmission of the to-be-released media data which is sent by the target page browser and is packaged based on the webpage real-time communication protocol to the target client based on the client media protocol, for example, the audio and video data generated by a browser user through a Web browser can be injected into a media server based on a private audio and video protocol through the proxy transmission channel, so that the problem of data exchange between the open-source webpage real-time communication protocol and the private audio and video protocol is solved, the media communication system applying the private audio and video protocol can realize the interconnection and intercommunication with the Web, and the interconnection and intercommunication between the Web and a mobile terminal and a local client installed on a PC (Personal Computer) terminal is realized.

Fig. 4 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure. As shown in fig. 4, compared with the above-mentioned fig. 2 embodiment, the method provided by the embodiment of the present disclosure is different in that the following steps may be further included.

In step S410, the media data to be subscribed, which is encapsulated by the client media protocol, is received from the target client through the proxy transmission channel.

The media data to be subscribed packaged by the client media protocol may be that the target client sends the media server first, and then the media server sends the media data to be subscribed packaged by the client media protocol to the proxy transmission channel in the streaming media server.

In the embodiment of the present disclosure, the media data to be subscribed refers to downlink media data, such as a downlink voice stream, a downlink video stream, and the like, currently prepared to be sent to a Web user or a browser user corresponding to a browsing end of a target page by a user at another end (for example, an audio and video user at another end) corresponding to a target client.

In an exemplary embodiment, the media transport channels may include media transport channels for subscription streams and the proxy transport channels may include proxy transport channels for publication streams.

Before receiving the media data to be subscribed encapsulated by the client media protocol from the target client through the proxy transmission channel, the method may further include: receiving a message for informing a new user to release a stream, which is initiated by a target page browsing terminal, through an agent transmission channel for releasing the stream; the proxy transmission channel for issuing the stream responds to inform a new user of issuing the stream message, generates a second producer role on the proxy transmission channel for issuing the stream, and stores a second producer role identifier of the second producer role; sending a second consumer role creation request message to a media transmission channel for a subscription stream through a proxy transmission channel for a publication stream; receiving and responding to the second consumer role creation request message through the media transmission channel for the subscription stream, and generating a second consumer role on the media transmission channel for the subscription stream; and subscribing the second consumer role to the second producer role identification through the media transmission channel for subscribing the stream to form a publish-subscribe relationship between the second consumer role and the second producer role. Further, a second consumer role identification for a second consumer role can also be saved.

In an exemplary embodiment, before receiving a message for notifying a new user of publishing a stream, which is initiated by a target page browser, through a proxy transmission channel for publishing a stream, the method may further include: receiving a second transmission channel creation request message initiated by a target page browsing terminal through a media stream processing process in a streaming media server; the media stream processing process responds to the second transmission channel creation request message, and a media transmission channel for subscribing the stream and an agent transmission channel for publishing the stream are created in the media stream processing process; and the media stream processing process generates an identifier of a media transmission channel for subscribing the stream and an identifier of a proxy transmission channel for publishing the stream, and returns the identifiers of the media transmission channel for subscribing the stream and the proxy transmission channel for publishing the stream to the target page browsing end.

In an exemplary embodiment, receiving the to-be-subscribed media data encapsulated in the client media protocol from the target client through the proxy transmission channel may include: and receiving the media data to be subscribed encapsulated by the client media protocol from the media server through the proxy transmission channel for publishing the stream. The media data to be subscribed, which is encapsulated by the client media protocol, is received by the media server from the target client.

In step S420, the media data to be subscribed, which is encapsulated by the client media protocol, is converted into the media data to be subscribed, which is encapsulated by the web communication media protocol, through the proxy transmission channel.

The converting, by the proxy transmission channel, the to-be-subscribed media data encapsulated by the client media protocol into the to-be-subscribed media data encapsulated by the web communication media protocol may include: analyzing the media data to be subscribed packaged by adopting a client media protocol by utilizing an agent transmission channel for publishing the stream; and the proxy transmission channel for publishing the stream converts the analyzed and obtained media data to be subscribed into the media data to be subscribed which is packaged by adopting a webpage communication media protocol.

In step S430, the media data to be subscribed, which is encapsulated by the web communication media protocol, is sent to the media transmission channel through the proxy transmission channel.

The sending, by the proxy transmission channel, the to-be-subscribed media data encapsulated by the web communication media protocol to the media transmission channel may include: and sending the media data to be subscribed, which is packaged by adopting a webpage communication media protocol, to a media transmission channel for subscribing the stream through a proxy transmission channel for publishing the stream based on the publishing and subscribing relationship between the second consumer role and the second producer role.

In step S440, the media data to be subscribed, which is encapsulated by the web communication media protocol, is sent to the target page browsing terminal through the media transmission channel.

The sending, through the media transmission channel, the media data to be subscribed, which is encapsulated by using the web communication media protocol, to the target page browsing terminal may include: and sending the media data to be subscribed, which is packaged by adopting a webpage communication media protocol, to a target page browsing end through a media transmission channel for subscribing the stream.

The media data processing method provided by the embodiment of the disclosure adds a proxy transmission channel based on a client media protocol in a streaming media server, when the proxy transmission channel based on the client media protocol in the streaming media server receives the to-be-subscribed media data encapsulated by the client media protocol from a media server, the proxy transmission channel can analyze the to-be-subscribed media data encapsulated by the client media protocol and convert the to-be-subscribed media data encapsulated by the client media protocol into the to-be-subscribed media data encapsulated by the web communication media protocol, and then transmits the to-be-subscribed media data encapsulated by the web communication media protocol to a media transmission channel based on the web communication media protocol by means of the proxy transmission channel, so that the media transmission channel based on the web communication media protocol can transmit the to-be-subscribed media data after protocol conversion to a target page browsing terminal, thereby realizing transmission of the to-be-subscribed media data encapsulated by the client media protocol and transmitted by a target client to the target page browsing terminal based on the web communication media protocol, that the to-be-subscribed media data generated by the target client can be injected into the web communication media protocol through the proxy transmission channel, and solving the problem of exchange between the media protocol and the client.

Fig. 5 schematically shows a flow chart of a media data processing method according to an embodiment of the present disclosure. In the embodiment of fig. 5, a process of transmitting downstream media data (i.e., media data to be subscribed) from a target client to a target page browsing end by using a media transmission channel for subscribing to a stream and a proxy transmission channel for publishing a stream in cooperation is described. In the embodiment of fig. 5, the web page communication media protocol is a web page real-time communication protocol, and the client media protocol is a private audio/video protocol.

It may be understood that, in the embodiment of the present disclosure, it is assumed that the media transmission channel and the proxy transmission channel are both simplex communication, and therefore, a media transmission channel for a publishing stream and a proxy transmission channel for a subscribing stream need to be adopted for transmission of uplink media data (to-be-published media data), and a media transmission channel for a subscribing stream and a proxy transmission channel for a publishing stream need to be adopted for transmission of downlink media data (to-be-subscribed media data), but the present disclosure is not limited thereto.

Or taking media data as audio and video data as an example, each target page browsing end creates two Peerconnection respectively for sending media data to be published and receiving media data to be subscribed, the sending end is used for sending local stream carrying local video track and audio track, and the receiving end receives remoteStream from other target clients.

As shown in fig. 5, compared with the above-mentioned fig. 3 embodiment, the method provided by the embodiment of the present disclosure may further include the following steps.

In step S501, a second transmission channel creation request message initiated by the target page browsing terminal is received through a media stream processing process in the streaming server.

The signaling server can also be used for responding to a join signaling sent by the target page browsing terminal, creating a user agent in the signaling server, and controlling the user agent to send a second transmission channel creation request message to the distributed message queue. The distributed message queue may also be used to send a second transmission channel creation request message to the scripting language execution environment. The scripting language runtime environment may also be used to send a second transmission channel creation request message to the media stream processing process.

In step S502, the media stream processing procedure responds to the second transmission channel creation request message, and creates a media transmission channel for subscribing to the stream and a proxy transmission channel for publishing the stream in the media stream processing procedure.

In step S503, the media stream processing process generates an identifier of a media transmission channel for subscribing to the stream and an identifier of a proxy transmission channel for publishing the stream, and returns them to the target page browsing terminal.

In step S504, a message for notifying the new user to publish the stream, which is initiated by the target page browser, is received through the proxy transmission channel for publishing the stream.

In an exemplary embodiment, the scripting language runtime environment in the streaming media server may be further configured to receive a notification new user to publish a streaming message from the distributed message queue and send the notification new user to publish the streaming message to the proxy transmission channel for subscribing to the streaming via interprocess communication.

The distributed message queue may also be used to receive a notification flow message from a user agent in the signaling server to notify a new user. The user agent can also be used for responding to a signaling for notifying the new user to publish the stream, which is sent by the target page browsing terminal, to generate a message for notifying the new user to publish the stream, and sending the message for notifying the new user to publish the stream to the distributed message queue.

In the embodiment of the present disclosure, the notification of a new user to publish a stream (AddNewUserTrack) message and the notification of a new user in a new user to publish a stream signaling refer to other users performing media data transmission with a browser user, for example, a mobile APP end user and/or a PC client user with which the browser user initiates a voice or video call, where the AddNewUserTrack message/signaling is used to notify an agent transmission channel to create a publish stream corresponding to the new user, so as to transmit media data published by the new user, that is, to-be-subscribed media data or downlink media data.

In step S505, the proxy transport channel for publishing the flow responds to notify the new user to publish the flow message, generates a second producer role on the proxy transport channel for publishing the flow, and saves a second producer role identifier of the second producer role.

In step S506, a second consumer role creation request message is transmitted to the media transmission channel for the subscription stream through the proxy transmission channel for the publish stream.

In step S507, a second consumer role is generated on the media transmission channel for the subscription stream by receiving and responding to the second consumer role creation request message through the media transmission channel for the subscription stream.

Further, a second consumer role identification for a second consumer role may also be saved.

In step S508, a publish-subscribe relationship between the second consumer role and the second producer role is formed by subscribing the second consumer role to the second producer role identifier through the media transmission channel for subscribing to the stream.

In step S509, the media data to be subscribed encapsulated by the private audio/video protocol is received from the media server through the proxy transmission channel for publishing the stream.

In step S510, the to-be-subscribed media data encapsulated by the private audio/video protocol is analyzed by using the proxy transmission channel for publishing the stream, and is converted into to-be-subscribed media data encapsulated by the web real-time communication protocol.

In step S511, based on the publish-subscribe relationship between the second consumer role and the second producer role, the media data to be subscribed, which is encapsulated by using the web real-time communication protocol, is sent to the media transmission channel for subscribing the stream through the proxy transmission channel for publishing the stream.

In step S512, the media data to be subscribed, which is encapsulated by the web real-time communication protocol, is sent to the target page browsing terminal through the media transmission channel for subscribing to the stream.

In the embodiment of the disclosure, by adding the proxy transmission channel to the streaming media server, a browser user of the WebRTC can be communicated with users of other ends adopting the client-side media protocol, and a function of exchanging media data, such as audio and video data, between users adopting different types of media protocols is realized. It is to be understood that, although the WebRTC gateway is used for exchanging media data between users of different types of media protocols in the above example, in practical applications, the WebRTC gateway may also be used for exchanging media data, such as audio and video data, between browser users of different webrtcs, but the above parsing and converting processes may be omitted.

According to the media data processing method provided by the embodiment of the disclosure, the transmission of the downlink media data from the target client to the target page browsing terminal can be realized through the media transmission channel for subscribing the stream and the proxy transmission channel for publishing the stream in the streaming media server, that is, the transmission of the media data to be subscribed sent by the target client and packaged based on the client media protocol to the target page browsing terminal based on the client media protocol is realized, and the problem of downlink media data exchange between the web communication media protocol and the client media protocol is solved.

In the embodiment of fig. 6, taking the web page communication media protocol as WebRTC as an example, a browser user logs in a target web page (e.g., a social or game web page) through a browser on a browser side of the target web page, where the browser may include the WebRTC module 621.

The streaming media server 610 can include a media streaming process (Worker) 611, which can include a media transport channel 6111 for published streams and a proxy transport channel 6112 for subscribed streams, as well as a media transport channel 6113 for subscribed streams and a proxy transport channel 6114 for published streams.

The browser user can send the media data to be published based on the WebRTC transmission protocol stack to the media transmission channel 6111 for publishing the stream through the WebRTC module 621 through an operation on the browser, and the media transmission channel 6111 for publishing the stream can send the media data to be published based on the WebRTC transmission protocol stack to the proxy transmission channel 6112 for subscribing the stream through the publishing and subscribing relationship between the first producer role in the media transmission channel 6111 for publishing the stream and the first consumer role in the proxy transmission channel 6112 for subscribing the stream, which is determined as described above. The proxy transmission channel 6112 for subscribing to the stream receives the media data to be published based on the WebRTC transmission protocol stack, analyzes the media data, repackages the media data to be published into media data to be published based on a client media protocol (here, taking a private audio/video protocol as an example), and sends the media data to the media server 630 for providing audio/video service, and the media server 630 communicates with audio/video users at other ends, such as the mobile APP end user 641 and the PC client end user 642, through the private audio/video protocol, so that the media data to be published can be played on a target client corresponding to the audio/video users at other ends.

An audio-video user at another end can send to-be-subscribed media data (for example, to-be-subscribed audio-video data) based on a private audio-video protocol to the media server 630 providing an audio-video service through the target client, the media server 630 can send the to-be-subscribed audio-video data based on the private audio-video protocol to the proxy transmission channel 6114 for publishing the stream, the proxy transmission channel 6114 for publishing the stream can analyze the to-be-subscribed audio-video data and convert the to-be-subscribed audio-video data into to-be-subscribed audio-video data based on the WebRTC transmission protocol stack, the proxy transmission channel 6114 for publishing the stream can send to-be-subscribed audio-video data based on the WebRTC transmission protocol stack to the media transmission channel 6113 for subscribing the stream through the publishing subscription relationship between the second producer role in the proxy transmission channel 6114 for publishing the stream and the second consumer role in the media transmission channel 6113 for subscribing the stream, and then the WebRTC transmission channel 6113 can transmit the to-be-subscribed audio-video data based on the rtc transmission protocol stack to the WebRTC transmission protocol stack to the target page browser user corresponding to the browser user.

The solution of MediaSoup in the related art is to use WebRTC transport protocol stack for audio-video communication between users. Its Router object can be abstracted into a Room under which many users can exchange audio and video data with each other. And the signaling layers of the two servers can be expanded on a MediaSoup NodesJS layer, the separation of signaling and media service is not realized, the dynamic capacity expansion or capacity contraction of the whole system is inconvenient, and once a streaming media server fails, the whole system can be unavailable. In the embodiment of fig. 7, a signaling server SigServer and a distributed message queue NATS are introduced, so that the above dynamic capacity expansion problem can be solved.

Fig. 7 schematically shows a schematic diagram of a media data transmission system according to an embodiment of the present disclosure. The embodiment of fig. 7 exemplifies that the method provided by the embodiment of the present disclosure is applied to a Web voice room of a game client, and when a browser user accesses the voice room of the game client on a target page browser, for example, an H5 page of a PC or a mobile phone, the browser user will provide the voice communication capability of the Web voice room.

The real-time voice room enables other users having the social relationship with the user to be added, and real-time voice is achieved within the user range corresponding to the social relationship of the virtual scene. However, the method provided by the embodiment of the present disclosure may be applied to any media data transmission scenario, for example, live webtv, video sharing, online radio, video on demand, video conference, online music on demand, and the like, but is not limited thereto.

As shown in fig. 7, the media data transmission system provided by the embodiment of the present disclosure may include a target page browsing terminal corresponding to a browser user, a streaming media service, and a game voice system providing a voice service. Fig. 7, fig. 8 and fig. 9 are illustrated by taking a media server as an example of a voice background server in a game voice system.

The browser at the target page browsing side may include a WebSDK module 622 and a WebRTC module 621. The WebSDK module 622 mainly interacts signaling with a signaling server (SigServer) 650 through WebSocket, and the WebRTC module 621 is mainly responsible for generating, receiving, and transmitting media streams.

Referring to fig. 7, the streaming media service part may include a signaling server (SigServer) 650, a distributed message queue (here, an example of NATS) 660, and a streaming media server. The streaming media server (taking mediasound as an example) is further divided into mediasound nodjs 612 and mediasound Worker 611 parts, which exchange information through interprocess communication.

In the fig. 7 embodiment, the role of SigServer 650 includes maintaining a signaling relationship with the browser user.

The SigServer 650 may be configured to maintain discoverable states of the mediasoups, where such messages may include heartbeat messages of the mediasoups, load information of the mediasoups (e.g., number of users on each MediaSoup, CPU (Central Processing Unit) utilization, memory transient data, etc.), and when the browser user sends the join signaling, the SigServer 650 may determine the mediasoups capable of normal communication according to the heartbeat messages of the mediasoups, and then select the most matching mediasoups from the normally-capable mediasoups according to the load information of the mediasoups to serve as the streaming media server for Processing the media data to be processed currently, for example, select the mediasoups with the lightest current load to serve the browser user, thereby playing a role of balancing.

The proxy transmission channels in the embodiment of fig. 7 may include the proxy transmission channel 6112 for subscribing to the stream and the proxy transmission channel 6114 for publishing the stream in the embodiment of fig. 6, the media transmission channel 6111 for publishing the stream and the media transmission channel 6113 for subscribing to the stream in the embodiment of fig. 6 may also be included in the streaming media server, the SigServer 650 does not process the stream (e.g., transmission of media data), and the SigServer 650 is used for delivering stream-related messages, and may include: the method comprises the steps of informing the streaming media server to create a first transmission channel creating request message of a media transmission channel for publishing the stream and a proxy transmission channel for subscribing the stream, informing the streaming media server to create a second transmission channel creating request message of the media transmission channel for subscribing the stream and the proxy transmission channel for publishing the stream, informing the media transmission channel for publishing the stream to create a first producer role creating request message, informing a new user of creating a second producer role by informing the proxy transmission channel for publishing the stream to publish the stream, receiving a publishing stream signaling sent by a browser user, receiving a subscribing stream signaling sent by the browser user, and informing the streaming media server to set a DTLS parameter.

NATS 660 acts as a low latency message queue, and in the embodiment of fig. 7 functions to pass messages between SigServer 650 and mediasound, maintaining a loosely coupled structure of SigServer 650 and mediasound.

Meanwhile, in the embodiment of fig. 7, the manner of separating signaling transmission and media data transmission makes mediasound and signaling service 650 in a loosely coupled relationship, so that mediasound or SigServer 650 can be dynamically added according to the load state of the whole system to support the normal operation of the service without stopping the whole system. And NATS 660 serves to bridge the communication between the signaling server 650 and the mediasound.

The MediaSoup NodeJS 612 layer can facilitate the expansion of MediaSoup and can conveniently utilize the huge tool library related to NodeJS. In the embodiment of the present disclosure, the mediasound nodess 612 layer is mainly used for transparently transmitting parameters between the NATS 660 and the media stream processing process Worker 611 of the streaming media server, and can also simply process some parameters.

The game voice system providing voice service mainly includes a voice background server 631 and SDK (Software Development Kit) parts of various clients (e.g., mobile APP and PC clients). The communication SDK for realizing the voice communication is implanted into the application program client side, the corresponding SDK is implanted into the background server of the application program, and the communication SDK in the application program client side and the SDK in the background server finish the audio communication through a private voice protocol.

In the embodiment of fig. 7, signaling transmission of WebRTC may be slow but needs to be reliable, while streaming media prioritizes real-time performance, and there may be packet loss and packet error, so a reliable transmission protocol based on TCP is used for transmission of signaling and the like, and an RTP protocol based on UDP (this protocol is taken as an example for illustration, but the disclosure is not limited thereto, and other protocols may also be used for transmission of real-time media data).

In order to ensure the security of media data transmission, webRTC may transmit media data through DTLS, where DTLS is an extension of TLS in UDP data transmission, and may be understood as an encrypted UDP data transmission channel. RTP corresponds to a format of data packets transmitted on DTLS.

In the embodiment of fig. 7, a user agent (client) is created in the signaling server 650, and the user agent is responsible for saving the user information of the browser user and controlling the media software Worker 611 to create a proxy transmission channel, which has the capability of understanding the proprietary voice protocol and the RTP data format, and the proxy transmission channel unpacks the media stream information in the RTP data (media data to be distributed) transmitted by the browser user WebRTC 621, repacks the media stream information by using the proprietary voice protocol, and forwards the media stream information to the game voice system providing the voice service. Meanwhile, when the proxy transmission channel receives the media data to be subscribed of the private voice protocol of the game voice system providing the voice service, the voice load of the media data is unpacked and repackaged into an RTP data packet to be forwarded to a browser end corresponding to the browser user for playing.

In the embodiment of fig. 7, in order to implement the intercommunication between the media data based on the private voice protocol and the media data based on the WebRTC in the mediasound, the concept of Router is simplified, the capability of broadcasting the media data and messages by the users in the Router is removed, each Web user and other users in the Router do not directly generate media data exchange any more, and they all receive the media data from the private voice background after passing through the private voice background. The media data channel of each user agent only publishes and subscribes the data stream to the agent data channel of the user agent, and the same agent data channel of the user agent only publishes and subscribes the data stream to the media data channel. The broadcasting of media data is implemented in private voice background (which can be extended to audio-video) rather than in Router.

The following describes an example of a process for implementing media data transmission by using the system provided in the embodiment of fig. 7 with reference to fig. 8 and 9.

Fig. 8 schematically shows a transmission timing diagram of an upstream media data stream according to an embodiment of the present disclosure. In the embodiment of fig. 8, the uplink media data stream is exemplified as an uplink voice stream.

As shown in fig. 8, in step S11, the browser user sends the join signaling of the browser user to the signaling server through the target page browser.

The join signaling in the embodiment of fig. 8 may carry user information of the browser user, for example, a user ID (which may be any information capable of uniquely identifying the browser user, such as a user name, an identity card number, a mobile phone number, and the like), a user login ticket, a roomld, and the like.

After receiving the join signaling, the signaling server triggers validity verification of the browser user, and if the verification is passed, a user agent is generated in the signaling server for storing relevant information of the browser user, for example, user information of the browser user is stored first. If the verification fails, the signaling server may return a join rejection notification message to the target page browsing end corresponding to the browser user.

The user agent is essentially a class object, which is an object that the browser user creates an object for identifying himself in the signaling server through the browser, and the subsequent browser user through the operation of the browser appears to operate the object to issue a corresponding command to the streaming media server, such as a selected mediasound.

In step S12, the signaling server sends signaling (here, a first transport channel creation message) to create a media transport channel and a proxy transport channel to the streaming server.

In step S12, after the user agent is successfully created, the signaling server transmits a first transmission channel creation message to the NATS through the user agent, the NATS transmits the first transmission channel creation message to the mediasound node js, and the mediasound node js sends the first transmission channel creation message to the mediasound Worker.

In step S13, the streaming media server sends a voice backend room entry signaling to the game voice backend server.

After the mediasound Worker receives the first transmission channel creation message, the mediasound sends a voice background room entering request signaling (joinroom signaling) to the game voice background server.

In step S14, the game voice backend server returns a successful room entry notification message to the streaming media server.

After receiving the join room signaling, the game voice background server authenticates the join room signaling, if the authentication is passed, a room entering success notification message is returned to the mediasound, and if the authentication is not passed, a room entering refusal notification message is returned to the mediasound. The inbound successful notification message may carry SDP information.

After receiving the room entry success notification message, the mediasound creates a media transmission channel for publishing the stream and a proxy transmission channel for subscribing the stream in the mediasound maker in response to the first transmission channel creation message, and generates an ID of the media transmission channel for publishing the stream and an ID of the proxy transmission channel for subscribing the stream.

In step S15, the streaming server returns a create success notification message to the signaling server.

After the media source Worker successfully creates the media transmission channel for publishing the stream and the proxy transmission channel for subscribing the stream, the media source Worker returns a creation success notification message to the signaling server through the media source node js and NATS, where the creation success notification message may carry the ID of the media transmission channel for publishing the stream, the ID of the proxy transmission channel for subscribing the stream, and the SDP information.

In step S16, the signaling server returns the SDP information and the ID of the transmission channel to the browser user.

After receiving the creation success notification message, the signaling server may return the ID of the media transmission channel for publishing the stream, the ID of the proxy transmission channel for subscribing to the stream, and the SDP information to the target page browser side corresponding to the browser user. The target page browser side can transmit the ID of the media transmission channel for publishing the stream and the ID of the proxy transmission channel for subscribing the stream to the user agent in the signaling server through WebSDK, so that the user agent also contains the ID of the media transmission channel for publishing the stream and the ID of the proxy transmission channel for subscribing the stream.

In step S17, the streaming server sends a voice handshake message to the game voice backend server.

For example, the mediasound sends a voice Hello message to the game voice backend server.

In step S18, the game voice background server returns a handshake success message to the streaming media server.

For example, if the game voice backend server returns a Hello success message to the mediasound, it indicates that the handshaking between the mediasound and the game voice backend server is successful.

In step S19, the browser user sends the release stream signaling to the signaling server.

For example, the browser user sends Publish signaling to the signaling server through the target page browsing terminal. The Publish signaling is to inform the mediasound to create a Producer, i.e. the first Producer role.

In step S110, the signaling server sends a signaling for creating a producer role (here, a first producer role creation request message) to the streaming server.

After receiving the Publish signaling, the signaling server sends a first producer role creation request message to the NATS through the user agent, the NATS sends the first producer role creation request message to the mediasound node js, the mediasound node js transparently transmits the first producer role creation request message to a media transmission channel for issuing a stream in the mediasound Worker, and after receiving the first producer role creation request message, the media transmission channel for issuing the stream creates a first producer role and stores a first producer role identifier.

In step S111, the streaming server returns a creation success message to the signaling server.

After the first producer role is successfully created in the mediasound, the mediasound Worker returns a creation success message to the signaling server through the mediasound nodess and NATS.

The media transmission channel (WebRTCTransportPub) for publishing the stream sends a first consumer role creation request message to a proxy transmission channel (Audio TransportSub) for subscribing the stream, the proxy transmission channel for subscribing the stream receives and responds to the first consumer role creation request message, a first consumer role is generated on the proxy transmission channel for subscribing the stream, and the first consumer role identification is stored, so that the first consumer role subscribes to a first producer role identification (ProducerID), and a publishing subscription relationship between the first consumer role and the first producer role is formed. So that the media data to be published at the Web browser can be transmitted to AudioTransportSub.

In step S112, the signaling server returns a release success message to the browser user.

And after receiving the creation success message in step S111, the signaling server returns a release success message to the target page browser side corresponding to the browser user.

In step S113, the browser user sends DTLS signaling to the signaling server.

The DTLS signaling is used to inform the opposite party of the encryption algorithm and key for data encryption at the Web browser side. SDP has partial fields that are encryption algorithms and keys that need padding data.

In step S114, the signaling server transmits a message for setting the DTLS parameter to the streaming server.

After receiving the DTLS signaling, the signaling server controls the user agent and sends a message for setting the DTLS parameter to the MediaSoup Worker through NATS and MediaSoup NodeJS in sequence.

In step S115, the streaming server returns a setting success message to the signaling server.

After receiving the message of setting the DTLS parameter, the media Soup Worker sets the DTLS parameter, and after the setting is successful, the media Soup NodeJS and NATS are used for returning the message of successful setting to the signaling server.

In step S116, the signaling server returns a setting success message to the browser user.

After receiving the setting success message returned in step S115, the signaling server returns the setting success message to the target page browser side corresponding to the browser user.

In step S117, the browser user sends signaling of the ICE creation session to the streaming server.

After receiving the setting success message returned in step S116, the target page browser sends signaling of ICE creation session to mediasound.

In step S118, the streaming server returns a message of media transmission channel establishment to the browser user.

After the mediasound receives the signaling of ICE to create the conversation, a transmission channel of media data between the browser and the mediasound is established as a media transmission channel, and after the establishment is successful, a message of establishing the media transmission channel is returned to a target page browsing end.

In step S119, the browser user sends the upstream media data stream to the streaming server.

After receiving the message established by the media transmission channel, the target page browsing terminal can send an upstream media data stream (i.e., media data to be distributed) to the mediasound.

In step S120, the streaming media server transmits the upstream media data stream to the game voice background server through the proxy transmission channel.

The media transmission channel for publishing the stream in the mediasound receives the upstream media data stream (here, the upstream voice stream is exemplified), and based on the publishing and subscribing relationship between the first producer role and the first consumer role, the media transmission channel for publishing the stream is forwarded to the proxy transmission channel for subscribing the stream, and the proxy transmission channel for subscribing the stream parses the stream, repackages the upstream voice stream into the upstream voice stream based on the private voice protocol, and then forwards the upstream voice stream to the game voice background server.

In the embodiment of the present disclosure, an object (media service object) called Audioserver is provided inside audioTransportSub, and is responsible for unpacking media data to be published based on webRTC from an RTP packet to obtain a media load, and repackaging the media load into media data to be published based on a private language protocol according to a private voice protocol.

A UDP data channel is maintained between the Audio Server object and the game voice background server, and a heartbeat protocol is kept alive, the channel is duplex, and the channel can be used for sending voice data of a Web browser end to the game voice background server and receiving the voice data from the game voice background server, unpacking the voice data into a voice load and then encapsulating the voice load into an RTP data packet.

As shown in fig. 8, when a browser user needs to publish a stream, the browser user needs to join in a streaming media service through join signaling, and in this process, a user agent of a signaling server, a media transmission channel for publishing the stream and a proxy transmission channel for subscribing the stream in the mediasound are created. Then through Publish and DTLS signaling, SDP information used by WebRTC for creating session is exchanged, the proxy data channel is responsible for analyzing RTP data flow transmitted by WebRTC and converting the RTP data flow into uplink voice flow of private voice protocol, and forwarding the uplink voice flow to the game voice background server, and the game voice background server can send the uplink voice flow to voice users at other ends.

Fig. 9 schematically shows a transmission timing diagram of a downstream media data stream according to an embodiment of the present disclosure. In the embodiment of fig. 9, the downlink media data stream is exemplified as a downlink voice stream.

As shown in fig. 9, in step S21, the browser user sends user join signaling to the signaling server.

Step S21 can refer to step S11 described above.

In step S22, the signaling server sends signaling (here, a second transmission channel creation message) to create the media transmission channel and the proxy transmission channel to the streaming server.

In step S22, after the user agent is successfully created, the signaling server transmits a second transmission channel creation message to the NATS through the user agent, the NATS retransmits the second transmission channel creation message to the mediasound node js, and the mediasound node js sends the second transmission channel creation message to the mediasound Worker.

In step S23, the streaming media server sends a voice backend room entry request signaling to the game voice backend service.

After the mediasound Worker receives the second transmission channel creation message, the mediasound Worker sends a voice background room entering request signaling (joinroom signaling) to the game voice background server.

In step S24, the game voice background service returns a notification message of successful room entry to the streaming media server.

After receiving the join room signaling, the game voice background server authenticates the join room signaling, if the authentication is passed, a room entering success notification message is returned to the mediasound, and if the authentication is not passed, a room entering refusal notification message is returned to the mediasound. The inbound success notification message may carry SDP information.

After receiving the successful room entry notification message, the mediasound creates a proxy transmission channel for publishing the stream and a media transmission channel for subscribing the stream in the mediasound maker in response to the second transmission channel creation message, and generates an ID of the proxy transmission channel for publishing the stream and an ID of the media transmission channel for subscribing the stream.

In step S25, the streaming server returns a create success notification message to the signaling server.

After the media sound Worker successfully creates the proxy transmission channel for publishing the stream and the media transmission channel for subscribing the stream, the media sound Worker returns a creation success notification message to the signaling server through the media sound nodjs and NATS, and the creation success notification message can carry the ID of the proxy transmission channel for publishing the stream, the ID of the media transmission channel for subscribing the stream, and the SDP information.

In step S26, the signaling server returns the SDP information and the ID of the transmission channel to the browser user.

After receiving the creation success notification message, the signaling server may return the ID of the proxy transmission channel for publishing the stream, the ID of the media transmission channel for subscribing the stream, and the SDP information to the target page browser side corresponding to the browser user. The target page browser side can transmit the ID of the proxy transmission channel for publishing the stream and the ID of the media transmission channel for subscribing the stream to the user agent in the signaling server through WebSDK, so that the user agent also contains the ID of the proxy transmission channel for publishing the stream and the ID of the media transmission channel for subscribing the stream.

In step S27, the streaming server sends a voice handshake message to the game voice backend server.

In step S28, the game voice backend server returns a handshake success message to the streaming media server.

Steps S27-S28 may refer to steps S17-S18 described above.

In step S29, the browser user sends a message (AddNewUserTrack signaling) to the signaling server notifying the new user of the release flow.

The new user in the new user issued stream message is notified, which may refer to a voice user at another end, and if the new user goes to the public, voice data may be sent out, and here, a transmission channel needs to be established in advance.

In step S210, the signaling server sends a message of proxy creation of a distribution stream to the streaming server.

After the signaling server receives the message informing the new user of releasing the stream, the user agent in the signaling server sends a message for the agent to create the release stream to the mediasound Worker through NATS and mediasound NodeJS, and informs the agent transmission channel for releasing the stream in the mediasound Worker to create the release stream. And the proxy transmission channel for publishing the stream responds to the message of creating the publishing stream by the proxy, generates a second producer role on the proxy transmission channel for publishing the stream, stores the second producer role identification and obtains a sender ID, wherein the sender ID is a new user identification and is used for identifying a new user publishing the media data to be subscribed.

In step S211, the streaming server returns a creation success notification message to the signaling server.

After the second producer role is successfully created, the mediasound returns a creation success notification message to the signaling server through NATS, and the creation success notification message carries the sender ID and the second producer role identifier.

In step S212, the signaling server returns the sender ID to the target page browsing end corresponding to the browser user.

In step S213, the browser user sends a subscription stream signaling to the signaling server.

The Subscribe signaling is used to inform the mediasound to create one consumer role, i.e. the second consumer role.

In step S214, the signaling server sends a message of creating a consumer role, here, a second consumer role creation request message, to the streaming server.

After receiving the Subscribe signaling, the signaling server sends a second consumer role creation request message to the mediasound through NATS, a media transmission channel for subscribing the stream responds to the second consumer role creation request message, a second consumer role is generated on the media transmission channel for subscribing the stream, and a second consumer role identifier of the second consumer role is stored, so that the second consumer role subscribes to the second producer role identifier, and a publishing and subscribing relationship between the second consumer role and the second producer role is formed.

The corresponding relation between the Producer and the concumer is established in the mediasound, and the received uplink voice stream and the downlink voice stream can be forwarded to the corresponding concumer through the Producer. And the process of creating the publication stream is to create a Producer. Producer corresponds to an object of a production flow, whose flow is transported based on Transport. The Consumer corresponds to an object consuming a flow, whose flow is also transported based on Transport.

When a browser user on a Web end needs to hear the voice of another user, an AddNewUserTrack signaling is sent first to notify the user agent to operate an AudioTransportPub (proxy transmission channel for publishing streams) to generate a Producer (second Producer role), then operate a WebRTCTransportSub (media transmission channel for subscribing streams) to generate a Consumer (second Consumer role), and then the Consumer subscribes to the Producer to form a publishing-subscribing relationship. One RTCPeerConnection can carry multiple tracks (media streams), so that an AddNewUserTrack signaling is reissued when a new user is on the microphone.

In step S215, the streaming server returns a create success message to the signaling server.

In step S216, the signaling server returns a success message to the browser user.

In step S217, the browser user sends DTLS signaling to the signaling server.

In step S218, the signaling server transmits a message for setting the DTLS parameter to the streaming server.

In step S219, the streaming server returns a setting success message to the signaling server.

In step S220, the signaling server returns a setting success message to the browser user.

In step S221, the browser user sends signaling of the ICE creation session to the streaming server.

In step S222, the streaming media server returns a message of media transmission channel establishment to the browser user.

Steps S215-S222 may refer to steps S111-S118 described above.

In the embodiments of fig. 8 and 9, the uplink and the downlink correspond to two Transport, specifically, two RTCPeerConnection are two independent connections at the Web browser end. Since the transmission channel here is simplex, the process of transmission channel establishment needs to be repeated.

In step S223, the streaming media server receives the downstream voice stream (the media data to be subscribed) from the game voice background server through the proxy transmission channel.

The mediasound receives the downstream voice stream from the game voice background server by using the proxy transmission channel for publishing the stream, analyzes and repacks the downstream voice stream into the downstream voice stream based on the WebRTC transmission protocol stack, and then sends the downstream voice stream to the media transmission channel for subscribing the stream.

In step S224, the streaming server sends the downstream media data stream to the browser user.

As shown in fig. 9, compared with fig. 8, the transmission process of the downlink voice is that a signaling for notifying an agent to create a publish stream is added, and after publishing a stream, a browser user actively subscribes to the publish stream to establish a complete publish-subscribe data path. After the downstream voice stream based on the private voice protocol is received by the proxy transmission channel, the downstream voice stream is unpacked and repackaged into an RTP data stream, and then forwarded to the browser user.

In the embodiment of the present disclosure, the AudioServer member of AudioTransport implements the logic of unpacking and packing the RTP packet and the private voice packet. The UDP data channel in AudioServer is duplex, and after receiving media data, it needs to find an agent transmission channel for publishing stream, i.e. audioTransportPub, in Router, and then transfers the media data to audioTransportPub, assuming that audioTransportPub receives network data, and actually the data of audioTransportPub does not come from the network directly, but comes from the UDP data channel of AudioServer. And finally conducting the data to the browser end of the browser user by using the publishing and subscribing relationship between the AudioTransportPub and the WebRTCTransportSub.

The media data processing method provided by the embodiment of the disclosure inherits the advantages of stability and easy expansion of a mediasound of an open-source streaming media server, carries out secondary development on the media data processing method, creates a proxy transmission channel on the proxy transmission channel through a user proxy, so that audio and video data generated by a Web browser can be injected into a background server corresponding to a client adopting a private audio and video protocol through the proxy transmission channel, and meanwhile, a Web end can also receive media data of the client adopting the private audio and video protocol through the proxy transmission channel, thereby solving the problem of data exchange between WebRTC communication and the private audio and video protocol, and realizing interconnection and intercommunication between a private audio and video communication system and the Web end.

Fig. 10 schematically shows a block diagram of a media data processing device according to an embodiment of the present disclosure. As shown in fig. 10, the media data processing apparatus 1000 provided by the embodiment of the present disclosure may be disposed in a streaming media server, and the streaming media server may include a media transmission channel and a proxy transmission channel. The media data processing device 1000 may include a to-be-published media data receiving unit 1010, a to-be-published media data forwarding unit 1020, a to-be-published media data converting unit 1030, and a to-be-published media data transmitting unit 1040.

In this embodiment of the present disclosure, the to-be-published media data receiving unit 1010 may be configured to receive, from the target web browser, to-be-published media data encapsulated by using a web communication media protocol through a media transmission channel. The to-be-published media data forwarding unit 1020 may be configured to send the to-be-published media data encapsulated by using the web communication media protocol to the proxy transmission channel through the media transmission channel. The to-be-published media data conversion unit 1030 may be configured to convert, by using the proxy transmission channel, to-be-published media data encapsulated by using a web communication media protocol into to-be-published media data encapsulated by using a client media protocol. The to-be-published media data sending unit 1040 is configured to send, through the proxy transmission channel, to-be-published media data encapsulated by the client media protocol to the target client.

In an exemplary embodiment, the media transport channels may include media transport channels for published streams and the proxy transport channels may include proxy transport channels for subscribed streams. The media data processing apparatus 1000 may further include: the first producer role creation request message receiving unit may be configured to receive, through the media transmission channel for the publishing stream, a first producer role creation request message initiated by the target page browsing terminal before receiving, through the media transmission channel, the to-be-published media data encapsulated by using the web communication media protocol from the target page browsing terminal; a first producer role generating unit, configured to generate a first producer role on a media transmission channel for publishing a stream in response to a first producer role creation request message, and store a first producer role identifier of the first producer role; a first consumer role creation request message sending unit, configured to send a first consumer role creation request message to a broker transmission channel for subscribing to a stream through a media transmission channel for publishing the stream; a first consumer role generation unit operable to receive and respond to the first consumer role creation request message through the proxy transmission channel for the subscription stream, and generate a first consumer role on the proxy transmission channel for the subscription stream; the first publish-subscribe relationship forming unit may be configured to subscribe the proxy transmission channel of the stream to enable the first consumer role to subscribe to the first producer role identifier, so as to form a publish-subscribe relationship between the first consumer role and the first producer role.

In an exemplary embodiment, the media data processing apparatus 1000 may further include: the first transmission channel creation request message receiving unit may be configured to receive, by a media stream processing process in the streaming media server, a first transmission channel creation request message initiated by a target page browsing terminal before receiving, by a media transmission channel for publishing a stream, a first producer role creation request message initiated by the target page browsing terminal; a first transmission channel creating unit, configured to respond to the first transmission channel creation request message by the media stream processing process, and create a media transmission channel for publishing a stream and a proxy transmission channel for subscribing to the stream in the media stream processing process; the first transmission channel identifier returning unit may be configured to generate, by the media stream processing process, an identifier of a media transmission channel used for publishing the stream and an identifier of a proxy transmission channel used for subscribing the stream, and return the identifier of the media transmission channel used for publishing the stream and the identifier of the proxy transmission channel used for subscribing the stream to the target page view port.

In an exemplary embodiment, the streaming media server may further include a scripting language execution environment, and the scripting language execution environment may be configured to receive the first producer role creation request message from the distributed message queue and send the first producer role creation request message to the media transmission channel for publishing the stream through inter-process communication.

The distributed message queue may be configured to receive a first producer role creation request message from a user agent in the signaling server, where the user agent contains user information of a target web page on a target page view, an identification of a media transmission channel for publishing a stream, and an identification of a proxy transmission channel for subscribing to the stream.

The user agent may be configured to generate a first producer role creation request message in response to a publish stream signaling sent by the target page browsing terminal, and send the first producer role creation request message to the distributed message queue.

In an exemplary embodiment, the signaling server may be configured to create a user agent in the signaling server in response to the join signaling sent by the target page view end, and control the user agent to send the first transmission channel creation request message to the distributed message queue.

The distributed message queue may also be used to send a first transport channel creation request message to the scripting language execution environment.

The scripting language runtime environment may also be used to send a first transport channel creation request message to the media stream processing process.

In an exemplary embodiment, the to-be-published media data receiving unit 1010 may include: the publishing streaming media transmission channel receiving unit may be configured to receive, from the target page browser, the to-be-published media data encapsulated by using the web communication media protocol through the media transmission channel for publishing the stream.

The to-be-published media data forwarding unit 1020 may include: the publishing streaming media transmission channel forwarding unit may be configured to send, based on a publishing subscription relationship between the first consumer role and the first producer role, the to-be-published media data encapsulated by using the web communication media protocol to the proxy transmission channel for subscribing to the stream through the media transmission channel for publishing the stream.

The to-be-published media data conversion unit 1030 may include: the subscription stream proxy transmission channel analysis unit can be used for analyzing the media data to be published which is packaged by adopting a webpage communication media protocol by utilizing a proxy transmission channel for subscription streams; the media data protocol conversion unit to be published can be used for the proxy transmission channel of the subscription stream to convert the media data to be published obtained by analysis into the media data to be published which is encapsulated by the media protocol of the client.

The to-be-published media data sending unit 1040 may include a to-be-published media data proxy sending unit, and may be configured to send, to the target client, the to-be-published media data encapsulated by using the client media protocol through a proxy transmission channel for subscribing to the stream.

In an exemplary embodiment, a media service object may be included in a proxy transport channel for a subscription stream, and a user datagram protocol data channel is maintained between the media service object and a media server. The to-be-published media data conversion unit 1030 may include: the media service object analyzing unit may be configured to analyze the to-be-published media data encapsulated by the web communication media protocol by using the media service object, and convert the to-be-published media data encapsulated by the web communication media protocol into the to-be-published media data encapsulated by the client media protocol.

The to-be-published media data sending unit 1040 may include: the user datagram protocol data channel transmission unit can be used for sending the to-be-released media data encapsulated by the client media protocol to the media server by using the user datagram protocol data channel so that the media server can transmit the to-be-released media data encapsulated by the client media protocol to the target client.

In an exemplary embodiment, the media data processing device 1000 may further include: the media data receiving unit to be subscribed can be used for receiving the media data to be subscribed, which is packaged by adopting a client media protocol, from the target client through the proxy transmission channel; the media data to be subscribed analyzing unit can be used for converting the media data to be subscribed, which is encapsulated by a client media protocol, into the media data to be subscribed, which is encapsulated by a web communication media protocol, through the proxy transmission channel; the device comprises a to-be-subscribed media data sending unit, a media transmission channel and a subscription unit, wherein the to-be-subscribed media data sending unit can be used for sending to-be-subscribed media data packaged by a webpage communication media protocol to the media transmission channel through the proxy transmission channel; the media data forwarding unit to be subscribed can be used for sending the media data to be subscribed, which is encapsulated by the web communication media protocol, to the target page browsing terminal through the media transmission channel.

In an exemplary embodiment, the media transport channels may include media transport channels for subscription streams and the proxy transport channels may include proxy transport channels for publication streams. The media data processing apparatus 1000 may further include: the new user notification issuing stream message receiving unit can be used for receiving a new user notification issuing stream message initiated by a target page browsing terminal through a proxy transmission channel for issuing streams before receiving the to-be-subscribed media data encapsulated by a client media protocol from a target client through the proxy transmission channel; a second producer role generating unit, configured to respond to the proxy transmission channel for issuing the stream to notify the new user of issuing the stream message, generate a second producer role on the proxy transmission channel for issuing the stream, and store a second producer role identifier of the second producer role; a second consumer role creation request message sending unit, configured to send a second consumer role creation request message to a media transmission channel for a subscription stream through a proxy transmission channel for a publish stream; the second consumer role generating unit may be configured to receive and respond to the second consumer role creation request message through the media transmission channel for the subscription stream, generate a second consumer role on the media transmission channel for the subscription stream, and store a second consumer role identifier of the second consumer role; the second publish-subscribe relationship forming unit may be configured to subscribe the second consumer role to the second producer role identifier through a media transmission channel for subscribing to the stream, so as to form a publish-subscribe relationship between the second consumer role and the second producer role.

In an exemplary embodiment, the to-be-subscribed media data receiving unit may include: and the publishing stream proxy transmission channel receiving unit may be configured to receive, from the media server through the proxy transmission channel for publishing the stream, the to-be-subscribed media data encapsulated by the client media protocol, where the to-be-subscribed media data encapsulated by the client media protocol is received by the media server from the target client.

The to-be-subscribed media data parsing unit may include: the publishing stream proxy transmission channel analysis unit can be used for analyzing the media data to be subscribed, which is encapsulated by the client media protocol, by using the proxy transmission channel for publishing the stream; the publishing stream proxy transmission channel converting unit may be configured to convert the media data to be subscribed, which is obtained through analysis, into the media data to be subscribed, which is encapsulated by the web communication media protocol, through the publishing stream proxy transmission channel.

The to-be-subscribed media data sending unit may include: and the publishing stream proxy transmission channel sending unit may be configured to send the media data to be subscribed, which is encapsulated by using the web communication media protocol, to the media transmission channel for subscribing the stream through the proxy transmission channel for publishing the stream based on a publishing and subscribing relationship between the second consumer role and the second producer role.

The to-be-subscribed media data forwarding unit may include: the subscription streaming media transmission channel sending unit may be configured to send the media data to be subscribed, which is encapsulated by using the web communication media protocol, to the target page browsing end through the media transmission channel for subscription streaming.

Other contents of the media data processing device of the embodiment of the present disclosure may refer to the above-described embodiment.

Further, the embodiment of the present disclosure also provides a media data transmission system, which may include a streaming media server, where the streaming media server may include a media transmission channel and a proxy transmission channel.

The media transmission channel can be used for receiving the to-be-published media data packaged by the webpage communication media protocol from the target webpage login terminal and sending the to-be-published media data packaged by the webpage communication media protocol to the proxy transmission channel. The proxy transmission channel can be used for converting the to-be-published media data encapsulated by the webpage communication media protocol into the to-be-published media data encapsulated by the client media protocol and sending the to-be-published media data encapsulated by the client media protocol to the target client login terminal.

In an exemplary embodiment, the system may further include a signaling server, which may include a user agent, wherein the user agent includes user information of the target web page logged on the target web page login terminal, an identification of the media transmission channel, and an identification of the agent transmission channel.

The user agent may be configured to receive a join signaling from the target web page login terminal, generate a transmission channel creation request message in response to the join signaling, and send the transmission channel creation request message to the distributed message queue, so that the transmission channel creation request message is sent to a media stream processing process in the streaming media server through the distributed message queue, where the media stream processing process creates a media transmission channel and a proxy transmission channel in the media stream processing process in response to the transmission channel creation request message.

In the embodiment of the present disclosure, the transmission channel creation request message may include a first transmission channel creation request message and a second transmission channel creation request message.

Other contents of the media data transmission system of the embodiment of the present disclosure may refer to the above-described embodiment.

It should be noted that although in the above detailed description several units of the device for action execution are mentioned, this division is not mandatory. Indeed, the features and functions of two or more of the units described above may be embodied in one unit, in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.

Reference is now made to fig. 11, which illustrates a schematic diagram of an electronic device suitable for use in implementing embodiments of the present application. The electronic device shown in fig. 11 is only an example, and should not bring any limitation to the functions and the use range of the embodiment of the present application.

Referring to fig. 11, an electronic device provided in an embodiment of the present disclosure may include: a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104.

Wherein the processor 1101, communication interface 1102 and memory 1103 are in communication with each other via a communication bus 1104.

Alternatively, the communication interface 1102 may be an interface of a communication module, such as an interface of a GSM (Global System for Mobile communications) module. The processor 1101 is used to execute programs. The memory 1103 is used for storing programs. The program may comprise a computer program comprising computer operating instructions. Wherein, can include in the procedure: and (4) a game client side program.

The processor 1101 may be a central processing unit CPU, or an Application Specific Integrated Circuit ASIC (Application Specific Integrated Circuit), or one or more Integrated circuits configured to implement embodiments of the present disclosure.

The memory 1103 may include a Random Access Memory (RAM) memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

Among them, the procedure can be specifically used for: receiving media data to be released, which is packaged by a webpage communication media protocol, from a target page browsing terminal through a media transmission channel in a streaming media server; sending the media data to be issued, which is encapsulated by adopting a webpage communication media protocol, to an agent transmission channel through a media transmission channel; converting the media data to be published packaged by the webpage communication media protocol into the media data to be published packaged by the client media protocol by using an agent transmission channel in the streaming media server; and sending the media data to be issued which is encapsulated by the client media protocol to a target client through the proxy transmission channel.

According to an aspect of the application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternative implementations of the embodiments described above.

It is to be understood that any number of elements in the drawings of the present disclosure are by way of example and not by way of limitation, and any nomenclature is used for differentiation only and not by way of limitation.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. The media data processing method is characterized in that the method is executed by a streaming media server, the streaming media server is a media package, the streaming media server comprises a media transmission channel, a proxy transmission channel, a script language operating environment and a media stream processing process, the script language operating environment is represented as media package node JS, the media stream processing process is a media package Worker, and the media package node JS and the media package Worker are communicated through processes;

the media transmission channel comprises a media transmission channel for publishing a stream, the proxy transmission channel comprises a proxy transmission channel for subscribing the stream, the proxy transmission channel for subscribing the stream comprises a media service object, and a user datagram protocol data channel is maintained between the media service object and a media server;

the signaling server is used for interacting with a target page browser end through WebSocket, responding to an adding signaling sent by the target page browser end, creating a user agent in the signaling server, and controlling the user agent to send a first transmission channel creation request message to a distributed message queue; the adding signaling carries user information of a target webpage; the user agent comprises user information of a target webpage on the target webpage browsing end, the identifier of the media transmission channel for publishing the stream and the identifier of the agent transmission channel for subscribing the stream;

the user agent is used for responding to a publishing stream signaling sent by the target page browsing end to generate a first producer role creating request message and sending the first producer role creating request message to the distributed message queue;

wherein the method comprises the following steps:

receiving the first transmission channel creation request message from the distributed message queue through the scripting language running environment, and sending the first transmission channel creation request message to the media stream processing process;

the media stream processing process responds to the first transmission channel creation request message, creates the media transmission channel for publishing streams and the proxy transmission channel for subscribing streams in the media stream processing process, generates an identifier of the media transmission channel for publishing streams and an identifier of the proxy transmission channel for subscribing streams, and returns the identifiers of the media transmission channel for publishing streams and the proxy transmission channel for subscribing streams to the target page browser;

receiving the first producer role creation request message from the distributed message queue through the scripting language running environment, and sending the first producer role creation request message to the media transmission channel for publishing the stream;

the media transmission channel for publishing the stream responds to the first producer role creation request message, generates a first producer role on the media transmission channel for publishing the stream, saves a first producer role identifier of the first producer role, and sends a first consumer role creation request message to the proxy transmission channel for subscribing the stream;

receiving and responding to the first consumer role creation request message through the proxy transmission channel for subscribing the stream, and generating a first consumer role on the proxy transmission channel for subscribing the stream, so that the first consumer role subscribes to the first producer role identifier, and a publish-subscribe relationship between the first consumer role and the first producer role is formed;

receiving the media data to be published packaged by adopting a webpage communication media protocol from the target page browser through the media transmission channel for publishing the stream;

based on the publishing and subscribing relation between the first consumer role and the first producer role, sending the to-be-published media data packaged by the webpage communication media protocol to the proxy transmission channel for subscribing the stream through the media transmission channel for publishing the stream;

analyzing the to-be-published media data packaged by the webpage communication media protocol by using the media service object in the proxy transmission channel for subscribing the stream, and converting the to-be-published media data into to-be-published media data packaged by a client media protocol, wherein the proxy transmission channel has a function of mutual conversion between the webpage communication media protocol and the client media protocol, the client media protocol is a protocol for realizing client media data communication, and the client media protocol comprises a private audio/video protocol adopted by a client;

and sending the media data to be published packaged by the client media protocol to a media server adopting the client media protocol through the proxy transmission channel for subscribing the stream and by utilizing the user datagram protocol data channel, wherein the media server sends the media data to be published packaged by the client media protocol to a target client.

2. The method of claim 1, wherein converting the to-be-published media data encapsulated by the web communication media protocol into the to-be-published media data encapsulated by the client media protocol by using the proxy transmission channel for the subscription stream comprises:

analyzing the media data to be published packaged by the webpage communication media protocol by using the proxy transmission channel for subscribing the stream;

and the proxy transmission channel for subscribing the stream converts the media data to be published obtained by analysis into the media data to be published packaged by adopting the client media protocol.

3. The method of claim 1, further comprising:

receiving the media data to be subscribed, which is packaged by the client media protocol, from the target client through the proxy transmission channel;

converting the media data to be subscribed, which is encapsulated by the client media protocol, into the media data to be subscribed, which is encapsulated by the webpage communication media protocol, through the proxy transmission channel;

sending the media data to be subscribed, which is packaged by the webpage communication media protocol, to the media transmission channel through the proxy transmission channel;

and sending the media data to be subscribed, which is packaged by the webpage communication media protocol, to the target page browsing terminal through the media transmission channel.

4. The method of claim 3, wherein the media transport channels comprise media transport channels for subscribing to streams, and wherein the proxy transport channels comprise proxy transport channels for publishing streams; wherein, before receiving the to-be-subscribed media data encapsulated in the client media protocol from the target client through the proxy transmission channel, the method further comprises:

receiving a message for notifying a new user to publish the stream, which is initiated by the target page browsing terminal, through the proxy transmission channel for publishing the stream;

the proxy transmission channel for issuing the stream responds to the message informing the new user of issuing the stream, generates a second producer role on the proxy transmission channel for issuing the stream, and stores a second producer role identifier of the second producer role;

sending a second consumer role creation request message to the media transmission channel for the subscription stream through the proxy transmission channel for the publish stream;

receiving and responding to the second consumer role creation request message through the media transmission channel for subscribing the stream, generating a second consumer role on the media transmission channel for subscribing the stream, and storing a second consumer role identifier of the second consumer role;

and subscribing the second consumer role to the second producer role identification through the media transmission channel for subscribing the stream to form a publish-subscribe relationship between the second consumer role and the second producer role.

5. The method of claim 4, wherein receiving the media data to be subscribed encapsulated in the client media protocol from the target client through the proxy transmission channel comprises:

receiving the media data to be subscribed, which is encapsulated by the client media protocol, from a media server through the proxy transmission channel for publishing the stream, wherein the media data to be subscribed, which is encapsulated by the client media protocol, is received by the media server from the target client;

converting the media data to be subscribed, which is encapsulated by the client media protocol, into the media data to be subscribed, which is encapsulated by the web communication media protocol, through the proxy transmission channel, and including:

analyzing the media data to be subscribed packaged by the client media protocol by using the proxy transmission channel for publishing the stream;

the proxy transmission channel for publishing the stream converts the media data to be subscribed obtained by analysis into the media data to be subscribed which is packaged by the webpage communication media protocol;

sending the media data to be subscribed, which is encapsulated by the web communication media protocol, to the media transmission channel through the proxy transmission channel, including:

based on the publishing and subscribing relationship between the second consumer role and the second producer role, sending the media data to be subscribed, which is packaged by the webpage communication media protocol, to the media transmission channel for subscribing the stream through the proxy transmission channel for publishing the stream;

the method for sending the media data to be subscribed, which is encapsulated by the webpage communication media protocol, to the target page browsing terminal through the media transmission channel comprises the following steps:

and sending the media data to be subscribed, which is packaged by the webpage communication media protocol, to the target page browsing terminal through the media transmission channel for subscribing the stream.

6. The media data processing device is arranged on a streaming media server, the streaming media server is a media sound, the streaming media server comprises a media transmission channel, a proxy transmission channel, a script language operating environment and a media stream processing process, the script language operating environment is represented as media sound NodeJS, the media stream processing process is media sound Worker, and the media sound NodeJS and the media sound Worker are communicated with each other through processes;

wherein the apparatus comprises:

a first transmission channel creation request message receiving unit, configured to receive the first transmission channel creation request message from the distributed message queue through the scripting language running environment, and send the first transmission channel creation request message to the media stream processing process;

a first transmission channel creating unit, configured to respond to the first transmission channel creation request message through the media stream processing process, and create the media transmission channel for publishing streams and the proxy transmission channel for subscribing to streams in the media stream processing process;

a first transmission channel identifier returning unit, configured to generate, through the media stream processing process, the identifier of the media transmission channel used for publishing the stream and the identifier of the proxy transmission channel used for subscribing the stream, and return the identifier of the media transmission channel used for publishing the stream and the identifier of the proxy transmission channel used for subscribing the stream to the target page browsing end;

a first producer role creation request message receiving unit, configured to receive the first producer role creation request message from the distributed message queue through the scripting language running environment, and send the first producer role creation request message to the media transmission channel for publishing the stream;

a first producer role generating unit, configured to respond to the first producer role creation request message through the media transmission channel for publishing the stream, generate a first producer role on the media transmission channel for publishing the stream, and store a first producer role identifier of the first producer role;

a first consumer role creation request message sending unit, configured to send a first consumer role creation request message to the proxy transmission channel for subscribing to the stream through the media transmission channel for publishing the stream;

a first consumer role generating unit, configured to receive and respond to the first consumer role creation request message through the proxy transmission channel for subscribing to the stream, and generate a first consumer role on the proxy transmission channel for subscribing to the stream;

a first publishing-subscribing relationship forming unit, configured to subscribe the first consumer role to the first producer role identifier through the proxy transmission channel for subscribing to the stream, and form a publishing-subscribing relationship between the first consumer role and the first producer role;

a to-be-published media data receiving unit, configured to receive, from the target page browsing end through the media transmission channel, to-be-published media data encapsulated by using a web communication media protocol; the media data receiving unit to be published comprises: a receiving unit of the distribution stream media transmission channel, configured to receive, from the target page browser, the to-be-distributed media data encapsulated by using the web communication media protocol through the media transmission channel for distributing the stream;

the media data forwarding unit to be issued is used for sending the media data to be issued, which is encapsulated by the webpage communication media protocol, to the proxy transmission channel through the media transmission channel; the media data forwarding unit to be issued comprises: a publishing stream media transmission channel forwarding unit, configured to send, based on a publishing subscription relationship between the first consumer role and the first producer role, to-be-published media data encapsulated by using the web communication media protocol through the media transmission channel for publishing a stream to the proxy transmission channel for subscribing the stream;

a to-be-published media data conversion unit, configured to convert, by using the proxy transmission channel for subscribing to the stream, to-be-published media data encapsulated by the web communication media protocol into to-be-published media data encapsulated by a client media protocol, where the proxy transmission channel has a function of interconversion between the web communication media protocol and the client media protocol, the client media protocol is a protocol for implementing client media data communication, and the client media protocol includes a private audio/video protocol adopted by a client; the media data conversion unit to be issued comprises: the media service object analysis unit is used for analyzing the to-be-issued media data encapsulated by the webpage communication media protocol by using the media service object and converting the to-be-issued media data encapsulated by the webpage communication media protocol into to-be-issued media data encapsulated by the client media protocol;

a to-be-published media data sending unit, configured to send to-be-published media data encapsulated by the client media protocol to a target client through the proxy transmission channel; the sending unit of the media data to be published comprises: and the user datagram protocol data channel transmission unit is used for sending the to-be-issued media data encapsulated by the client media protocol to a media server adopting the client media protocol by using the user datagram protocol data channel so that the media server can transmit the to-be-issued media data encapsulated by the client media protocol to the target client.

7. The apparatus of claim 6, wherein the to-be-published media data conversion unit comprises:

the subscription stream proxy transmission channel analysis unit is used for analyzing the media data to be published which is encapsulated by the webpage communication media protocol by using the proxy transmission channel for the subscription stream;

and the to-be-published media data protocol conversion unit is used for converting the to-be-published media data obtained by analysis into the to-be-published media data packaged by the client media protocol through the proxy transmission channel for subscribing the stream.

8. The apparatus of claim 6, further comprising:

a to-be-subscribed media data receiving unit, configured to receive, from the target client through the proxy transmission channel, to-be-subscribed media data encapsulated by using the client media protocol;

the media data to be subscribed is analyzed by the media data to be subscribed, which is encapsulated by the client media protocol, through the proxy transmission channel and is converted into the media data to be subscribed, which is encapsulated by the web communication media protocol;

a to-be-subscribed media data sending unit, configured to send, through the proxy transmission channel, to-be-subscribed media data encapsulated by the web communication media protocol to the media transmission channel;

and the to-be-subscribed media data forwarding unit is used for sending the to-be-subscribed media data packaged by the webpage communication media protocol to the target page browsing terminal through the media transmission channel.

9. The apparatus of claim 8, wherein the media transport channels comprise media transport channels for subscribing to streams, and wherein the proxy transport channels comprise proxy transport channels for publishing streams; wherein the apparatus further comprises:

a new user notification issuing stream message receiving unit, configured to receive a new user notification issuing stream message initiated by the target page browser through the proxy transmission channel for issuing a stream before receiving, from the target client through the proxy transmission channel, to-be-subscribed media data encapsulated by the client media protocol;

a second producer role generating unit, configured to respond to the notification new user issue stream message through the proxy transmission channel for issuing a stream, generate a second producer role on the proxy transmission channel for issuing a stream, and store a second producer role identifier of the second producer role;

a second consumer role creation request message sending unit, configured to send a second consumer role creation request message to the media transmission channel for subscribing to the stream through the proxy transmission channel for publishing the stream;

a second consumer role generating unit, configured to receive and respond to the second consumer role creation request message through the media transmission channel for subscribing to the stream, generate a second consumer role on the media transmission channel for subscribing to the stream, and store a second consumer role identifier of the second consumer role;

and a second publish-subscribe relationship forming unit, configured to subscribe the second consumer role to the second producer role identifier through the media transmission channel for subscribing to the stream, so as to form a publish-subscribe relationship between the second consumer role and the second producer role.

10. The apparatus of claim 9, wherein the to-be-subscribed media data receiving unit comprises:

a receiving unit of a published stream proxy transmission channel, configured to receive, from a media server through the proxy transmission channel for publishing streams, to-be-subscribed media data encapsulated by using the client media protocol, where the to-be-subscribed media data encapsulated by using the client media protocol is received by the media server from the target client;

wherein, the media data analysis unit to be subscribed comprises:

the publishing stream proxy transmission channel analysis unit is used for analyzing the media data to be subscribed which is encapsulated by the client media protocol by using the proxy transmission channel for publishing the stream;

the publishing stream proxy transmission channel conversion unit is used for converting the media data to be subscribed acquired by analysis into the media data to be subscribed packaged by the webpage communication media protocol through the proxy transmission channel for publishing the stream;

wherein, the to-be-subscribed media data sending unit comprises:

a publishing stream proxy transmission channel sending unit, configured to send, based on a publishing subscription relationship between the second consumer role and the second producer role, to-be-subscribed media data encapsulated by using the web communication media protocol through the proxy transmission channel for publishing a stream to the media transmission channel for subscribing the stream;

wherein, the media data forwarding unit to be subscribed includes:

and the subscription streaming media transmission channel sending unit is used for sending the to-be-subscribed media data packaged by the webpage communication media protocol to the target page browsing end through the media transmission channel for subscription streaming.

11. A media data transmission system is characterized in that the system comprises a streaming media server, a signaling server and a distributed message queue;

the streaming media server is a mediasound, the streaming media server comprises a media transmission channel, a proxy transmission channel, a script language operating environment and a media stream processing process, the script language operating environment is represented as mediasound nodeJS, the media stream processing process is mediasound Worker, and the mediasound nodeJS and the mediasound Worker are communicated with each other through processes;

the signaling server is used for interacting with a target page browsing terminal through a WebSocket, responding to a join signaling sent by the target page browsing terminal, creating a user agent in the signaling server, and controlling the user agent to send a first transmission channel creation request message to a distributed message queue; the adding signaling carries user information of a target webpage; the user agent comprises user information of a target webpage on the target webpage browsing end, the identifier of the media transmission channel for publishing the stream and the identifier of the agent transmission channel for subscribing the stream;

the script language operating environment is used for receiving the first transmission channel creation request message from the distributed message queue and sending the first transmission channel creation request message to the media stream processing process;

the media stream processing process is configured to create, in response to the first transmission channel creation request message, the media transmission channel for publishing streams and the proxy transmission channel for subscribing to streams in the media stream processing process, generate an identifier of the media transmission channel for publishing streams and an identifier of the proxy transmission channel for subscribing to streams, and return the identifier of the media transmission channel for publishing streams and the identifier of the proxy transmission channel for subscribing to streams to the target page browser;

the script language operating environment is further configured to receive the first producer role creation request message from the distributed message queue and send the first producer role creation request message to the media transmission channel for publishing the stream;

the media transmission channel for publishing the stream is used for responding to the first producer role creation request message, generating a first producer role on the media transmission channel for publishing the stream, and storing a first producer role identifier of the first producer role; sending a first consumer role creation request message to the proxy transmission channel for subscribing to the stream;

the proxy transmission channel for the subscription stream is used for receiving and responding to the first consumer role creation request message and generating a first consumer role on the proxy transmission channel for the subscription stream; enabling the first consumer role to subscribe to the first producer role identification, and forming a publish-subscribe relationship between the first consumer role and the first producer role;

the media transmission channel for publishing the stream is also used for receiving the to-be-published media data packaged by adopting a webpage communication media protocol from the target page browser terminal, and sending the to-be-published media data packaged by adopting the webpage communication media protocol to the proxy transmission channel for subscribing the stream based on the publishing and subscribing relation between the first consumer role and the first producer role;

the proxy transmission channel for subscribing the stream is further used for analyzing the to-be-published media data packaged by the webpage communication media protocol by using the media service object in the proxy transmission channel, converting the to-be-published media data packaged by the client media protocol into the to-be-published media data packaged by the client media protocol, and sending the to-be-published media data packaged by the client media protocol to a media server adopting the client media protocol by using the user datagram protocol data channel, wherein the media server sends the to-be-published media data packaged by the client media protocol to a target client login terminal, the proxy transmission channel has a function of mutual conversion between the webpage communication media protocol and the client media protocol, the client media protocol is a protocol for realizing client media data communication, and the client media protocol comprises a private audio and video protocol adopted by a client.

12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 5.

13. An electronic device, comprising: at least one processor; a storage device configured to store at least one program that, when executed by the at least one processor, causes the at least one processor to implement the method of any one of claims 1 to 5.