CN114640893A - Method and device for connecting service end of real-time audio and video communication system - Google Patents

Method and device for connecting service end of real-time audio and video communication system Download PDF

Info

Publication number
CN114640893A
CN114640893A CN202210234617.4A CN202210234617A CN114640893A CN 114640893 A CN114640893 A CN 114640893A CN 202210234617 A CN202210234617 A CN 202210234617A CN 114640893 A CN114640893 A CN 114640893A
Authority
CN
China
Prior art keywords
stream
push
pull
real
communication system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210234617.4A
Other languages
Chinese (zh)
Inventor
李庆波
王孝庆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jingdong Technology Information Technology Co Ltd
Original Assignee
Jingdong Technology Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jingdong Technology Information Technology Co Ltd filed Critical Jingdong Technology Information Technology Co Ltd
Priority to CN202210234617.4A priority Critical patent/CN114640893A/en
Publication of CN114640893A publication Critical patent/CN114640893A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/633—Control signals issued by server directed to the network components or client
    • H04N21/6332—Control signals issued by server directed to the network components or client directed to client
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/654—Transmission by server directed to the client

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Security & Cryptography (AREA)
  • Telephonic Communication Services (AREA)

Abstract

The application discloses a method and a device for connecting a service end of a real-time audio and video communication system, which can be applied to a proxy end of the real-time audio and video communication system, wherein the method comprises the following steps: responding to a push-pull stream interface calling instruction sent by a server, and sending a push-pull stream path establishment request to a real-time audio and video communication system; establishing a push-pull stream path between the real-time audio and video communication system and a target media server based on the push-pull stream path establishing request; and performing push-pull stream data transmission between the server and the target media server based on the push-pull stream path. The method and the device can simplify the process of butt joint of the push-pull stream module of the server and the push-pull stream of the real-time audio and video communication system, reduce the cost of butt joint and access, and reduce the difficulty of butt joint development.

Description

Method and device for connecting service end of real-time audio and video communication system
Technical Field
The present application relates to the field of data processing, and in particular, to a method and an apparatus for performing a service end connection in a real-time audio/video communication system, a computer device, and a storage medium.
Background
In a Real-Time audio/video Communication (WebRTC) system, a common scenario is to pull and push streams from and to a Real-Time audio/video system at a server. In the related art, push-pull streaming is generally performed through a standard WebRTC access procedure. However, with the standard WebRTC access, a connection mechanism established by ICE (Interactive Connectivity Establishment) is complicated, for example, a communication party needs to collect Candidate (a data structure describing a connection endpoint in the WebRTC) and perform Candidate exchange under the coordination of signaling service. In addition, the standard WebRTC push-pull stream docking has high cost and great docking development difficulty.
Disclosure of Invention
The application provides a method and a device for butt joint of a server side of a real-time audio and video communication system, computer equipment and a storage medium, which can simplify the butt joint process of a push-pull stream module of the server side and a push-pull stream of the real-time audio and video communication system, reduce the butt joint and access cost and reduce the butt joint development difficulty.
According to a first aspect of the present application, a method for a server side of a real-time audio and video communication system to be connected is provided, which is applied to a proxy side of the real-time audio and video communication system, and the method includes:
responding to a push-pull stream interface calling instruction sent by the server, and sending a push-pull stream path establishment request to the real-time audio and video communication system;
establishing a push-pull stream path between the real-time audio and video communication system and a target media server based on the push-pull stream path establishing request;
and performing push-pull stream data transmission between the server and the target media server based on the push-pull stream path.
According to a second aspect of the present application, there is provided a server side docking device of a real-time audio and video communication system, which is applied to a proxy side of the real-time audio and video communication system, the device includes:
the relay endpoint module is used for responding to a push-pull stream interface calling instruction sent by the server and sending a push-pull stream path establishment request to the real-time audio and video communication system;
the relay endpoint module is also used for establishing a push-pull stream path between the relay endpoint module and a target media server in the real-time audio and video communication system based on the push-pull stream path establishing request;
and the data transmission module is used for performing push-pull stream data transmission between the server side and the target media server based on the push-pull stream path.
According to a third aspect of the present application, there is provided a computer device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method for service end interfacing of a real-time audio-video communication system according to the first aspect.
According to a fourth aspect of the present application, there is provided a computer-readable storage medium, where the computer instructions are configured to cause the computer to execute the server-side interfacing method of the real-time audio/video communication system according to the foregoing first aspect.
According to a fifth aspect of the present application, there is provided a computer program product comprising a computer program which, when executed by a processor, performs the steps of the method of the aforementioned first aspect.
According to the technical scheme of the application, the push-pull stream path is established and maintained through the agent end of the real-time audio and video communication system, and the process of butt joint of the push-pull stream module of the server end and the push-pull stream of the real-time audio and video communication system is simplified. In addition, the server side can complete push-pull stream by calling the interface in the agent side of the real-time audio and video communication system, the complexity of the system behind the server side is shielded, the development cost of docking and accessing the real-time audio and video communication system is greatly reduced, the docking development difficulty is reduced, and the docking and accessing cost is reduced.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
fig. 1 is a flowchart of a method for terminating a server in a real-time audio/video communication system according to an embodiment of the present application;
fig. 2 is a flowchart of another method for terminating a server in a real-time audio/video communication system according to an embodiment of the present application;
fig. 3 is a flowchart of a server side interfacing method of a real-time audio/video communication system according to an embodiment of the present application;
FIG. 4 is an interaction illustration of a docking system provided in accordance with an embodiment of the present application;
fig. 5 is a server docking device of a real-time audio/video communication system according to an embodiment of the present application;
FIG. 6 illustrates a schematic block diagram of an example computer device that can be used to implement embodiments of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application to assist in understanding, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a flowchart of a method for terminating a server in a real-time audio/video communication system according to an embodiment of the present application. It should be noted that the service end connection method of the real-time audio and video communication system in the embodiment of the present application can be applied to a proxy end of the real-time audio and video communication system. That is to say, the service end connection method of the real-time audio/video communication system according to the embodiment of the present application is described from the agent end side of the real-time audio/video communication system.
As shown in fig. 1, the service-end-to-end method may include, but is not limited to, the following steps.
In step 101, a push-pull stream interface call instruction sent by a server is responded, and a push-pull stream path establishment request is sent to a real-time audio and video communication system.
Optionally, the server may send a push-pull stream interface call instruction to the agent of the real-time audio/video communication system. And the real-time audio and video communication system agent end can send a push-pull stream path establishment request to the real-time audio and video communication system when receiving the push-pull stream interface calling instruction sent by the server end.
For example, the server side is provided with a push-pull stream module, and the real-time audio and video communication system agent side can provide a push-pull stream interface. The push-pull flow module in the server side can start a push-pull flow process by calling a push-pull flow interface provided by the agent side of the real-time audio and video communication system, and at the moment, a push-pull flow path establishment request can be sent to the real-time audio and video communication system.
In step 102, a push-pull stream path is established with a target media server in the real-time audio-video communication system based on the push-pull stream path establishment request.
Optionally, when a push-pull stream path establishment request is sent to the real-time audio-video communication system, the real-time audio-video communication system proxy may establish a push-pull stream path between the server and the real-time audio-video communication system based on the push-pull stream path establishment request.
In step 103, push-pull stream data transmission between the server and the target media server is performed based on the push-pull stream path.
Optionally, push-pull streaming data transmission between the server and a target media server in the real-time audio/video communication system may be implemented based on the push-pull stream path. For example, a stream pushing path between the server and the real-time audio/video communication system is established through the real-time audio/video communication system proxy, and the real-time audio/video communication system proxy can push audio/video stream data in the server to a corresponding stream receiving media server in the real-time audio/video communication system through the stream pushing path. For another example, a pull stream path between the server and the real-time audio/video communication system is established through the real-time audio/video communication system proxy, and the real-time audio/video communication system proxy can read audio/video stream data of a stream to be pulled from a target media server in the real-time audio/video communication system through the pull stream path and send the audio/video stream data to the server, so that push-pull stream interfacing between the server and the real-time audio/video communication system is realized.
According to the method for the butt joint of the server side of the real-time audio and video communication system, the push-pull flow path can be established and maintained through the proxy side of the real-time audio and video communication system, and the process of butt joint of the push-pull flow module of the server side and the push-pull flow of the real-time audio and video communication system is simplified. In addition, the server side can complete push-pull stream by calling the interface in the agent side of the real-time audio and video communication system, the complexity of the system behind the server side is shielded, the development cost of docking and accessing the real-time audio and video communication system is greatly reduced, the docking development difficulty is reduced, and the docking and accessing cost is reduced.
It should be noted that the push-pull flow process in the embodiment of the present application includes a push-pull flow process and a pull flow process. Optionally, when the agent of the real-time audio and video communication system receives that the push-pull stream interface call instruction sent by the server is the pull stream interface call instruction, it may be considered that the server wants to perform pull stream docking with the real-time audio and video communication system. When the agent end of the real-time audio and video communication system receives the push-pull stream interface calling instruction sent by the server end as a push-pull stream interface calling starting instruction, the agent end of the real-time audio and video communication system can think that the server end wants to carry out push-stream docking with the real-time audio and video communication system. The process of performing pull stream docking and push stream docking between the server and the real-time audio/video communication system will be described with reference to the drawings.
Fig. 2 is a flowchart of another service port connection method of a real-time audio-video communication system according to an embodiment of the present application. It should be noted that the service end connection method according to the embodiment of the present application may be described from an agent side of a real-time audio/video communication system. As shown in fig. 2, the service-end-to-end method may include, but is not limited to, the following steps.
In step 201, a pull stream interface call instruction sent by the server is responded, and a pull stream path establishment request is sent to the real-time audio and video communication system.
Optionally, the server may send a pull stream interface call instruction to the agent of the real-time audio/video communication system. And the real-time audio and video communication system agent end can send a pull stream path establishment request to the real-time audio and video communication system when receiving the pull stream interface calling instruction sent by the server end.
For example, the service end is provided with a pull stream module, and the real-time audio and video communication system agent end can provide a pull stream interface. The stream pulling module in the server can start stream pulling by calling a stream pulling interface provided by the agent end of the real-time audio and video communication system, for example, calling a video stream pulling interface to start to pull a video stream, and calling an audio stream interface to start to pull an audio stream. At this time, the agent end of the real-time audio and video communication system can call the route establishing interface of the relay endpoint RelayIO module of the agent end to start sending a pull path establishing request to the real-time audio and video communication system.
In step 202, a target flow path is obtained based on the pull flow path setup request.
Optionally, the relay endpoint RelayIO module at the real-time audio/video communication system proxy end may obtain the target stream path from the scheduling module.
In step 203, a route establishing request is sent to a target media server corresponding to the target stream path so as to establish a pull stream path with the target media server.
Optionally, the relay endpoint RelayIO module of the real-time audio/video communication system proxy may establish a path according to the target flow path and the RelayIO module of the radial target media server, and maintain the path state.
In step 204, based on the pull stream path, the audio/video stream data of the stream to be pulled corresponding to the server is read from the target media server.
Optionally, when the creation of the pull flow path is completed, the target media server needs to create a related object, such as sendfile (Transport module), provider (Consumer), etc., to form audio/video stream data of the to-be-pulled stream, where the audio/video stream data may be SDP (session description Protocol), RTP (Real-time Transport Protocol) data.
In this embodiment, the real-time audio/video communication system proxy may read audio/video stream data of the stream to be pulled, such as SDP and RTP data of the stream to be pulled, from the target media server.
In step 205, the audio/video stream data is analyzed to obtain audio/video frame data, and the audio/video frame data is sent to the server.
Optionally, the audio/video stream data of the stream to be pulled may be analyzed to obtain audio/video frame data of the stream, and the audio/video frame data is sent to the server, so that the server sends the audio/video frame data to the corresponding client for audio/video playing.
In order to reduce the delay, optionally, in some embodiments of the present application, before parsing the audio/video stream data, the audio/video stream data may be subjected to network anti-jitter processing. For example, when the agent of the real-time audio/video communication system reads audio/video stream data of a stream to be pulled from the target media server, the agent may perform network anti-jitter processing on the audio/video stream data, and then may perform analysis processing on the audio/video stream data subjected to the network anti-jitter processing to obtain audio/video frame data of the stream, so as to send the audio/video frame data to the server.
In order to further avoid the delay, optionally, in some embodiments of the present application, when a pull stream is established, a plurality of pull stream paths may be created between a pull stream module in a server and a real-time audio/video communication system, and a proxy of the real-time audio/video communication system detects a delay on each pull stream path and selects a pull stream path with a low delay for data transmission.
According to the method for the service end connection of the real-time audio and video communication system, the pull flow path can be established and maintained through the proxy end of the real-time audio and video communication system, and the process of the pull flow connection between the pull flow module of the service end and the real-time audio and video communication system is simplified. In addition, the server side can finish stream pulling by calling the interface in the agent side of the real-time audio and video communication system, the complexity of the system behind is shielded, the development cost of docking and accessing the real-time audio and video communication system is greatly reduced, the docking development difficulty is reduced, and the docking and accessing cost is reduced.
Fig. 3 is a flowchart of a service end connection method of a real-time audio/video communication system according to an embodiment of the present application. It should be noted that the server-side connection method according to the embodiment of the present application may be described from a proxy side of a real-time audio/video communication system. As shown in fig. 3, the server side docking method may include, but is not limited to, the following steps.
In step 301, in response to a start stream pushing interface call instruction sent by the server, address information of the target stream receiving media server is obtained.
Optionally, the server may send a call instruction for starting a stream pushing interface to the agent of the real-time audio/video communication system. When receiving the call instruction of the start stream pushing interface sent by the server, the agent of the real-time audio/video communication system may obtain Address information of the target stream receiving media server, where the Address information may include an IP (Internet Protocol Address) Address and a port number of the target stream receiving media server.
For example, the server has a stream pushing module, and the real-time audio/video communication system agent can provide a stream pushing starting interface. The stream pushing module in the server can start stream pushing by calling a stream pushing starting interface provided by the real-time audio and video communication system agent, for example, calling a video stream pushing starting interface to start pushing a video stream, and calling an audio stream pushing starting interface to start pushing an audio stream. At this time, the agent end of the real-time audio and video communication system can call the open interface getPublishWorker to obtain the address information of the target streaming media receiving server. The target streaming media receiving server can be understood as a media server for receiving audio and video streams pushed by a server in a real-time audio and video communication system.
In step 302, a stream pushing path establishment request is sent to the real-time audio/video communication system according to the address information.
Optionally, the real-time audio/video communication system proxy may call a stream pushing path establishment interface of the relayIO module of the home terminal to send a stream pushing path establishment request to the real-time audio/video communication system, so as to start establishing a stream pushing path to the real-time audio/video communication system.
In step 303, based on the push flow path establishment request, a route establishment data packet is sent to the target stream receiving media server, so that the target stream receiving media server executes a stream receiving preparation operation when receiving the route establishment data packet.
Optionally, when the routing data packet reaches a target stream receiving media server in the real-time audio/video communication system, the target stream receiving media server starts to create a stream receiving related object, and creates a related object of a media layer stream receiving, so that the target stream receiving media server performs a stream receiving preparation operation.
In step 304, a notification message sent by the target streaming media receiving server is received, where the notification message is used to notify the real-time audio/video communication system that the creation of the streaming path at the proxy end is completed.
Optionally, after the streaming preparation work is completed, the target streaming media receiving server notifies the real-time audio and video communication system that the creation of the streaming path is completed at the proxy end. And the real-time audio and video communication system agent end can confirm that the creation of the stream pushing path is completed when receiving the notification message sent by the target stream receiving media server.
In step 305, in response to a stream pushing interface calling instruction sent by the server, pushing audio and video stream data to the target stream receiving media server based on the stream pushing path.
Optionally, the real-time audio and video communication system proxy may call back a stream pushing module in the server to notify the stream pushing module that the creation of the stream pushing path is completed. The server calls the stream pushing interface to really push the audio and video data to the real-time audio and video communication system, for example, calls the video stream pushing interface to send the video data, and calls the audio stream pushing interface to send the audio data.
Optionally, in some embodiments of the present application, when a path is established in a push stream, multiple push stream paths may be created between a push stream module in a server and a real-time audio/video communication system, and a proxy of the real-time audio/video communication system detects a delay on each push stream path and selects a push stream path with a low delay for data transmission.
According to the method for the service end connection of the real-time audio and video communication system, the plug flow path can be established and maintained through the agent end of the real-time audio and video communication system, and the plug flow connection process of the service end plug flow module and the real-time audio and video communication system is simplified. In addition, the server side can complete plug flow by calling the interface in the agent side of the real-time audio and video communication system, the complexity of the system behind the server side is shielded, the development cost of butt joint and access to the real-time audio and video communication system is greatly reduced, the butt joint development difficulty is reduced, and the butt joint and access cost is reduced.
In order to facilitate a clearer understanding of the present application by those skilled in the art, the following detailed description will be made with reference to fig. 4.
As shown in fig. 4, the docking system may be used to implement docking of a server with a real-time audio/video communication system. The docking system may include a service end 410, a real-time audio and video communication system agent end 420, and a real-time audio and video communication system 430. As an example, the real-time audio/video communication system agent may be integrated on the server, or the real-time audio/video communication system agent and the server are respectively deployed, which is not specifically limited in this application.
Alternatively, as shown in fig. 4, the server 410 has a push-pull flow module 411 therein, and the push-pull flow module 411 may include a push-pull flow module and a pull flow module. The real-time audio/video communication system agent end 420 may include an SDK (Software Development Kit) module 421, a relay endpoint RelayIO module 422, and an anti-jitter module 423. The real-time audio/video communication system 430 includes a plurality of media servers, and in fig. 4, for example, a certain media server, the media server may include a control module 431, a RelayIO module 432, and a signaling module 433. The real-time audio-video communication system 430 may include a scheduler 41 and an open interface 42. The real-time audio/video communication system agent 420 is a bridge for interfacing the push-pull stream module 411 and the real-time audio/video communication system 430, and the real-time audio/video communication system agent 420 has the functions of simplifying the flow of the push-pull stream and reducing the access cost of the push-pull stream. The details of the pull and push flows are described below in conjunction with the system architecture diagram of fig. 4.
The pull logic is as follows:
1) the pull flow module calls the pull flow interface of the SDK module 421 to start the pull flow. For example, calling the audio stream pull interface starts pulling the video stream, and calling the video stream pull interface starts pulling the audio stream.
2) The SDK module 421 calls the route establishing interface of the RelayIO module 422 at the local end to start establishing a pull path to the real-time audio/video communication system 430.
3) The RelayIO module 422 of the SDK module 421 acquires the target flow path from the scheduler 41.
4) The RelayIO module 422 of the SDK module 421 establishes a path to the RelayIO module 432 of the target media server side according to the target flow path, and maintains the path state.
5) The control module 431 in the target media server creates a relevant object (e.g., sendtransportport, concumer, etc.), and sends the streamed SDP, RTP data to the RelayIO module 432.
6) The RelayIO module 422 of the SDK module 421 reads the SDP and RTP data of the stream from the RelayIO module 432 at the opposite end, and outputs the SDP and RTP data to the SDK module 421 through a callback function.
7) The SDK module 421 sends the SDP and RTP data of the stream to the anti-jitter module 423 for network anti-jitter processing.
8) The SDK module 421 unpacks the data processed by the anti-jitter module 423 into frames (Frame data), and submits the data to the pull flow module through a callback function.
The push flow logic is as follows:
1) the push flow module calls the start push flow interface of the SDK module 421 to start the push flow. For example, calling the video stream start plug interface starts to push the video stream, and calling the audio stream start plug interface starts to push the audio stream.
2) The SDK module 421 calls the open interface 42 to obtain the address information of the target streaming media server, such as the IP address and the port number.
3) The SDK module 421 calls a stream pushing path establishing interface of the local RelayIO module 422 to start establishing a stream pushing path to the real-time audio/video communication system 430.
4) The RelayIO module 422 of the SDK module 421 establishes a path to the RelayIO module 432 of the target streaming media server, and maintains a path state.
5) When the routing packet reaches the target streaming media receiving server of the real-time audio/video communication system 430, the control module 431 in the target streaming media receiving server notifies the signaling module 433 to start creating a streaming related object.
6) The signaling module 433 informs the control module 431 to create the related object of the media layer received stream.
7) After the completion of the streaming preparation work of the target streaming media server, the relay io module 422 of the SDK module 421 is notified that the creation of the streaming path is completed.
8) The SDK module 421 calls back the push flow module push flow path creation completion.
9) The stream pushing module calls a stream pushing interface to really push the audio and video data to the real-time audio and video communication system 430. For example, a video stream plug interface is called to transmit video data, and an audio stream plug interface is called to transmit audio data.
In the embodiment of the application, when the push-pull stream is established, a plurality of paths are actually created between the push-pull stream module and the real-time audio and video communication system, the relay io module detects delay on each path, and selects a path with low delay for data transmission.
Fig. 5 is a server docking device of a real-time audio/video communication system according to an embodiment of the present application. It should be noted that the service end connection device of the real-time audio/video communication system may be applied to a proxy end of the real-time audio/video communication system. As shown in fig. 5, the service-side docking device 500 may include: a relay endpoint module 510 and a data transmission module 520.
The relay endpoint module 510 is configured to send a request for establishing a push-pull stream path to the real-time audio/video communication system in response to a push-pull stream interface call instruction sent by the server.
The relay endpoint module 510 is further configured to establish a push-pull stream path with a target media server in the real-time audio-video communication system based on the push-pull stream path establishment request.
The data transmission module 520 is configured to perform push-pull stream data transmission between the server and the target media server based on the push-pull stream path.
In some embodiments of the present application, the push-pull flow interface call instruction may be a push-pull flow interface call instruction. In this embodiment, the relay endpoint module 510 is specifically configured to: and responding to a pull stream interface calling instruction sent by the server, and sending a pull stream path establishing request to the real-time audio and video communication system.
In this embodiment of the present application, the relay endpoint module 510 is specifically configured to: acquiring a target flow path based on the pull flow path establishment request; and sending a route establishing request to a target media server corresponding to the target stream path so as to establish a pull stream path with the target media server.
In this embodiment of the present application, the relay endpoint module 510 is further configured to read, from the target media server, audio and video stream data of a stream to be pulled corresponding to the server, based on the pull stream path; the data transmission module 520 is specifically configured to: and analyzing the audio and video stream data to obtain audio and video frame data, and sending the audio and video frame data to the server.
In one implementation, on the basis as shown in fig. 5, the service-end connection device may further include: and an anti-jitter module. The anti-jitter module is used for carrying out network anti-jitter processing on the audio and video stream data before the data transmission module analyzes and processes the audio and video stream data.
In some embodiments of the present application, the push-pull flow interface call instruction may be a start push-pull flow interface call instruction; in this embodiment, the relay endpoint module 510 is specifically configured to: responding to a starting stream pushing interface calling instruction sent by a server side, and acquiring address information of a target stream receiving media server; and sending a stream pushing path establishing request to a real-time audio and video communication system according to the address information.
In this embodiment of the present application, the relay endpoint module 510 is specifically configured to: based on the stream pushing path establishing request, sending a path establishing data packet to a target stream receiving media server, and enabling the target stream receiving media server to execute stream receiving preparation work when receiving the path establishing data packet; and receiving a notification message sent by the target streaming media receiving server, wherein the notification message is used for notifying the real-time audio and video communication system agent end of completing the creation of the streaming path.
In this embodiment of the present application, the data transmission module 520 is specifically configured to: and responding to a stream pushing interface calling instruction sent by the server, and pushing audio and video stream data to the target stream receiving media server based on the stream pushing path.
With regard to the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.
According to the server side docking device of the real-time audio and video communication system, the push-pull stream path is established and maintained through the proxy side of the real-time audio and video communication system, and the process of docking the push-pull stream module of the server side with the push-pull stream of the real-time audio and video communication system is simplified. In addition, the server side can complete push-pull stream by calling the interface in the agent side of the real-time audio and video communication system, the complexity of the system behind the server side is shielded, the development cost of docking and accessing the real-time audio and video communication system is greatly reduced, the docking development difficulty is reduced, and the docking and accessing cost is reduced.
Based on the embodiment of the application, the application also provides a computer device, at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the service end connection method of the real-time audio and video communication system according to any one of the foregoing embodiments.
Based on the embodiment of the present application, a computer-readable storage medium is further provided, where the computer instructions are configured to enable a computer to execute the method for server-side connection of the real-time audio/video communication system according to any one of the foregoing embodiments provided in the embodiment of the present application.
FIG. 6 shows a schematic block diagram of an example computer device that can be used to implement embodiments of the present application. Computer devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The computing unit 601 performs the various methods and processes described above, such as the server-side interfacing method of a real-time audio-video communication system. For example, in some embodiments, the service-side interfacing method of a real-time audio-video communication system may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the service end interfacing method of the real-time audio-video communication system described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured in any other suitable way (e.g., by means of firmware) as a server-side interfacing method of a real-time audio-video communication system.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), the internet, and blockchain networks.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service extensibility in a traditional physical host and VPS service ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server incorporating a blockchain.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
The above-described embodiments are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims (19)

1. A method for connecting a service end of a real-time audio and video communication system is characterized in that the method is applied to a proxy end of the real-time audio and video communication system and comprises the following steps:
responding to a push-pull stream interface calling instruction sent by the server, and sending a push-pull stream path establishment request to the real-time audio and video communication system;
establishing a push-pull stream path between the real-time audio and video communication system and a target media server based on the push-pull stream path establishing request;
and performing push-pull stream data transmission between the server and the target media server based on the push-pull stream path.
2. The method of claim 1, wherein the push-pull flow interface call instruction is a push-pull flow interface call instruction; the sending a push-pull stream path establishment request to the real-time audio and video communication system in response to the push-pull stream interface call instruction sent by the server side comprises the following steps:
and responding to the pull stream interface calling instruction sent by the server, and sending a pull stream path establishing request to the real-time audio and video communication system.
3. The method according to claim 2, wherein the establishing a push-pull stream path with a target media server in the real-time audio-video communication system based on the push-pull stream path establishment request comprises:
acquiring a target flow path based on the pull flow path establishing request;
and sending a route establishing request to a target media server corresponding to the target stream path so as to establish a pull stream path with the target media server.
4. The method of claim 3, wherein the performing push-pull streaming data transmission between the server and the target media server based on the push-pull streaming path comprises:
based on the pull stream path, reading audio and video stream data of the stream to be pulled corresponding to the server from the target media server;
and analyzing the audio and video stream data to obtain audio and video frame data, and sending the audio and video frame data to the server.
5. The method according to claim 4, wherein before said parsing said audio-video streaming data, said method further comprises:
and carrying out network anti-jitter processing on the audio and video stream data.
6. The method of claim 1, wherein the push-pull flow interface call instruction is a start push-flow interface call instruction; the sending a push-pull stream path establishment request to the real-time audio and video communication system in response to the push-pull stream interface call instruction sent by the server side comprises the following steps:
responding to the call instruction of the stream pushing starting interface sent by the server, and acquiring address information of a target stream receiving media server;
and sending a stream pushing path establishing request to the real-time audio and video communication system according to the address information.
7. The method of claim 6, wherein establishing the push-pull flow path with a target media server in the real-time audio-video communication system based on the push-pull flow path establishment request comprises:
based on the stream pushing path establishment request, sending a path establishment data packet to the target stream receiving media server, and enabling the target stream receiving media server to execute stream receiving preparation work when receiving the path establishment data packet;
and receiving a notification message sent by the target streaming media receiving server, wherein the notification message is used for notifying the real-time audio and video communication system agent end that the creation of the streaming path is completed.
8. The method of claim 7, wherein the performing push-pull streaming data transmission between the server and the target media server based on the push-pull streaming path comprises:
and responding to a stream pushing interface calling instruction sent by the server, and pushing audio and video stream data to the target stream receiving media server based on the stream pushing path.
9. A service end docking device of a real-time audio and video communication system is characterized in that the device is applied to a proxy end of the real-time audio and video communication system and comprises:
the relay endpoint module is used for responding to a push-pull stream interface calling instruction sent by the server and sending a push-pull stream path establishment request to the real-time audio and video communication system;
the relay endpoint module is also used for establishing a push-pull stream path between the relay endpoint module and a target media server in the real-time audio and video communication system based on the push-pull stream path establishing request;
and the data transmission module is used for performing push-pull stream data transmission between the server side and the target media server based on the push-pull stream path.
10. The apparatus of claim 9, wherein the push-pull flow interface call instruction is a push-pull flow interface call instruction; the relay endpoint module is specifically configured to:
and responding to the pull stream interface calling instruction sent by the server, and sending a pull stream path establishing request to the real-time audio and video communication system.
11. The apparatus of claim 10, wherein the relay endpoint module is specifically configured to:
acquiring a target flow path based on the pull flow path establishing request;
and sending a route establishing request to a target media server corresponding to the target stream path so as to establish a pull stream path with the target media server.
12. The apparatus of claim 11,
the relay endpoint module is further configured to read audio and video stream data of a stream to be pulled corresponding to the server from the target media server based on the pull stream path;
the data transmission module is specifically configured to: and analyzing the audio and video stream data to obtain audio and video frame data, and sending the audio and video frame data to the server.
13. The apparatus of claim 12, further comprising:
and the anti-jitter module is used for carrying out network anti-jitter processing on the audio and video stream data before the data transmission module analyzes and processes the audio and video stream data.
14. The apparatus of claim 9, wherein the push-pull flow interface call command is a start push-pull flow interface call command; the relay endpoint module is specifically configured to:
responding to the call instruction of the stream pushing starting interface sent by the server, and acquiring address information of a target stream receiving media server;
and sending a stream pushing path establishing request to the real-time audio and video communication system according to the address information.
15. The apparatus of claim 14, wherein the relay endpoint module is specifically configured to:
based on the stream pushing path establishment request, sending a path establishment data packet to the target stream receiving media server, and enabling the target stream receiving media server to execute stream receiving preparation work when receiving the path establishment data packet;
and receiving a notification message sent by the target streaming media receiving server, wherein the notification message is used for notifying the real-time audio and video communication system agent end that the creation of the streaming path is completed.
16. The apparatus of claim 15, wherein the data transmission module is specifically configured to:
and responding to a stream pushing interface calling instruction sent by the server, and pushing audio and video stream data to the target stream receiving media server based on the stream pushing path.
17. A computer device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of service end interfacing for a real-time audio-visual communications system as claimed in any one of claims 1 to 8.
18. A computer-readable storage medium, wherein the computer instructions are configured to cause the computer to execute the method for terminating the service end of the real-time audio-video communication system according to any one of claims 1 to 8.
19. A computer program product comprising a computer program which, when executed by a processor, carries out the steps of the method of any one of claims 1 to 8.
CN202210234617.4A 2022-03-10 2022-03-10 Method and device for connecting service end of real-time audio and video communication system Pending CN114640893A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210234617.4A CN114640893A (en) 2022-03-10 2022-03-10 Method and device for connecting service end of real-time audio and video communication system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210234617.4A CN114640893A (en) 2022-03-10 2022-03-10 Method and device for connecting service end of real-time audio and video communication system

Publications (1)

Publication Number Publication Date
CN114640893A true CN114640893A (en) 2022-06-17

Family

ID=81948711

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210234617.4A Pending CN114640893A (en) 2022-03-10 2022-03-10 Method and device for connecting service end of real-time audio and video communication system

Country Status (1)

Country Link
CN (1) CN114640893A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115134628A (en) * 2022-06-27 2022-09-30 深圳市欢太科技有限公司 Streaming media transmission method, device, terminal equipment and storage medium

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106941629A (en) * 2017-04-05 2017-07-11 深圳进门财经科技股份有限公司 Real-time live broadcast method based on SIP+RTP Yu RTMP protocol interconnections
CN108696772A (en) * 2017-04-11 2018-10-23 上海谦问万答吧云计算科技有限公司 A kind of transmission method and device of real-time video
CN110417877A (en) * 2019-07-17 2019-11-05 视联动力信息技术股份有限公司 A kind of processing method and processing device of audio-video business
CN111314786A (en) * 2019-12-10 2020-06-19 杭州当虹科技股份有限公司 PC screen sharing method based on WebRTC
CN111818360A (en) * 2020-09-14 2020-10-23 平安国际智慧城市科技股份有限公司 Media on-demand method, system and device
CN112235595A (en) * 2020-11-04 2021-01-15 无锡蓝软智能医疗科技有限公司 Operation scene dynamic presenting method, system and medium based on mixed reality technology
CN112423146A (en) * 2019-08-21 2021-02-26 亦非云互联网技术（上海）有限公司 Playing data acquisition and pushing method, system, medium, server and playing terminal
CN112533006A (en) * 2020-11-05 2021-03-19 深圳市咪码科技有限公司 Communication method and device for live broadcast platform and VOIP terminal
CN112788053A (en) * 2021-01-25 2021-05-11 深圳市欢太科技有限公司 Real-time communication method, device, server, system and storage medium
CN112929592A (en) * 2021-02-07 2021-06-08 聚好看科技股份有限公司 Video call method, display device and server
CN112933611A (en) * 2021-03-26 2021-06-11 厦门雅基软件有限公司 Cloud game progress sharing method and device and electronic equipment
CN113824808A (en) * 2020-06-21 2021-12-21 慧与发展有限责任合伙企业 Method and system for network address translation traversal using an intermediary encounter agent

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106941629A (en) * 2017-04-05 2017-07-11 深圳进门财经科技股份有限公司 Real-time live broadcast method based on SIP+RTP Yu RTMP protocol interconnections
CN108696772A (en) * 2017-04-11 2018-10-23 上海谦问万答吧云计算科技有限公司 A kind of transmission method and device of real-time video
CN110417877A (en) * 2019-07-17 2019-11-05 视联动力信息技术股份有限公司 A kind of processing method and processing device of audio-video business
CN112423146A (en) * 2019-08-21 2021-02-26 亦非云互联网技术（上海）有限公司 Playing data acquisition and pushing method, system, medium, server and playing terminal
CN111314786A (en) * 2019-12-10 2020-06-19 杭州当虹科技股份有限公司 PC screen sharing method based on WebRTC
CN113824808A (en) * 2020-06-21 2021-12-21 慧与发展有限责任合伙企业 Method and system for network address translation traversal using an intermediary encounter agent
CN111818360A (en) * 2020-09-14 2020-10-23 平安国际智慧城市科技股份有限公司 Media on-demand method, system and device
CN112235595A (en) * 2020-11-04 2021-01-15 无锡蓝软智能医疗科技有限公司 Operation scene dynamic presenting method, system and medium based on mixed reality technology
CN112533006A (en) * 2020-11-05 2021-03-19 深圳市咪码科技有限公司 Communication method and device for live broadcast platform and VOIP terminal
CN112788053A (en) * 2021-01-25 2021-05-11 深圳市欢太科技有限公司 Real-time communication method, device, server, system and storage medium
CN112929592A (en) * 2021-02-07 2021-06-08 聚好看科技股份有限公司 Video call method, display device and server
CN112933611A (en) * 2021-03-26 2021-06-11 厦门雅基软件有限公司 Cloud game progress sharing method and device and electronic equipment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115134628A (en) * 2022-06-27 2022-09-30 深圳市欢太科技有限公司 Streaming media transmission method, device, terminal equipment and storage medium
CN115134628B (en) * 2022-06-27 2024-08-20 深圳市欢太科技有限公司 Streaming media transmission method, streaming media transmission device, terminal equipment and storage medium

Similar Documents

Publication Publication Date Title
US10498784B2 (en) Method for an audio/video live broadcast in an HTML5-based browser
US10601878B2 (en) Call processing method and control apparatus, automatic call distribution apparatus, and agent terminal
US10764435B2 (en) Smart customer assistant in a contact center
US20200280591A1 (en) System and Method for Efficient Port and Bandwidth Utilization in Setting up Communication Sessions
KR100985612B1 (en) Automatic orchestration of dynamic multiple party, multiple media communications
EP2779579A1 (en) Method, device, and system for realizing voip call in cloud computing environment
CN103348657B (en) Flow media playing method, equipment and system
CN107846379B (en) Port multiplexing method and server in video conference system
CN115473876A (en) Method, device and system for transmitting real-time streaming media data and storage medium
CN114640893A (en) Method and device for connecting service end of real-time audio and video communication system
US9961120B2 (en) System and method for managing concurrent communications in a contact center
CN116033235B (en) Data transmission method, digital person production equipment and digital person display equipment
CN114710471B (en) Customer service voice communication method and device based on network, electronic equipment and medium
US9059860B2 (en) Techniques for announcing conference attendance changes in multiple languages
CN103166938A (en) Voice over internet protocol (VOIP) streaming media data transmission method, device and system based on cloud computing framework
CN114401254B (en) Streaming media service processing method and device, electronic equipment and storage medium
CN113727183B (en) Live push method, apparatus, device, storage medium and computer program product
CN109150856A (en) Realize method, system, electronic equipment and the storage medium of videoconference
CN112738026B (en) Commanding and scheduling method
CN112671970B (en) Control method and control device for mobile equipment and cloud mobile phone, electronic equipment, mobile equipment, cloud server and medium
CN113992637B (en) Audio and video data receiving method, device, equipment, system and storage medium
CN118042214A (en) Screen projection method, device and system
CN118741041A (en) Video conference method and device based on relay function, electronic equipment and medium
CN114615236A (en) WebRTC-based multi-user audio and video communication method
CN118264650A (en) Browser real-time communication method, system, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination