WO2023197897A1 - 直播音视频流的处理方法、装置、设备及介质 - Google Patents

直播音视频流的处理方法、装置、设备及介质 Download PDF

Info

Publication number
WO2023197897A1
WO2023197897A1 PCT/CN2023/085866 CN2023085866W WO2023197897A1 WO 2023197897 A1 WO2023197897 A1 WO 2023197897A1 CN 2023085866 W CN2023085866 W CN 2023085866W WO 2023197897 A1 WO2023197897 A1 WO 2023197897A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio
live
client
video stream
video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/085866
Other languages
English (en)
French (fr)
Inventor
廖南山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Priority to US18/856,859 priority Critical patent/US20250260846A1/en
Publication of WO2023197897A1 publication Critical patent/WO2023197897A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2368—Multiplexing of audio and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting

Definitions

  • the present disclosure relates to the technical field of video processing, and in particular to a processing method, device, equipment and medium for live audio and video streams.
  • live broadcast has gradually emerged.
  • one client of the live broadcast can serve as the inviter to invite another live broadcast client for a live broadcast connection.
  • the other live broadcast client allows the live broadcast connection, audio and video calls between the two live broadcast rooms can be realized, thus realizing A live interactive effect.
  • the present disclosure provides a processing method, device, equipment and medium for live audio and video streams.
  • Embodiments of the present disclosure provide a method for processing live audio and video streams.
  • the method is applied to a first client and includes the following steps: in response to a live broadcast connection operation with a second client, sending a live broadcast directed to the second client. connection request; receiving the second live audio and video stream corresponding to the second client; in response to the received live broadcast connection pass message corresponding to the second client, playing the combined audio and video stream, wherein the combined audio and video stream is
  • the video stream is obtained by converging the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client, wherein the live connection connection message is for the Pass message of the live broadcast connection request.
  • Embodiments of the present disclosure provide a method for processing live audio and video streams.
  • the method is applied to a second client and includes the following steps: in response to a received live broadcast connection request from the first client, obtaining the first client The corresponding first live audio and video stream; in response to the live connection confirmation operation, feedback the live connection pass message, and play the combined audio and video stream, wherein the combined audio and video stream is based on the first direct message corresponding to the first client.
  • the broadcast video stream and the second live audio and video stream corresponding to the second client are merged together, wherein the live broadcast connection confirmation operation is a confirmation connection operation for the live broadcast connection request, and the live broadcast connection passes a message for the live broadcast connection request via message.
  • Embodiments of the present disclosure also provide a device for processing live audio and video streams.
  • the device is used on a first client and includes: a sending module, configured to send a message to the third client in response to a live broadcast connection operation with the second client.
  • the live broadcast connection request of the two clients; the first receiving module is used to receive the second live audio and video stream corresponding to the second client; the first combined stream playback module is used to respond to the received second live audio and video stream of the second client.
  • the corresponding live broadcast connection plays a combined audio and video stream through a message, wherein the combined audio and video stream is based on the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client.
  • the video streams are obtained by converging processing, wherein the live broadcast connection pass message is a pass message for the live broadcast connection request.
  • Embodiments of the present disclosure also provide a device for processing live audio and video streams.
  • the device is used on a second client and includes: a second receiving module configured to obtain, in response to a received live broadcast connection request from the first client.
  • the feedback module is used to respond to the live broadcast connection confirmation operation and feedback the live broadcast connection pass message;
  • the second merge playback module is used to respond to the live broadcast connection confirmation operation and play the merged stream Audio and video streams, wherein the combined audio and video stream is obtained by converging the first live audio and video stream corresponding to the first client and the second live audio and video stream corresponding to the second client, wherein , the live broadcast connection confirmation operation is a confirmation connection operation for the live broadcast connection request, and the live broadcast connection pass message is a pass message for the live broadcast connection request.
  • An embodiment of the present disclosure also provides an electronic device.
  • the electronic device includes: a processor; a memory used to store instructions executable by the processor; and the processor is used to read the instruction from the memory.
  • the instructions can be executed, and the instructions are executed to implement the processing method for live audio and video streams as provided in the embodiments of the present disclosure.
  • Embodiments of the present disclosure also provide a computer-readable storage medium, the storage medium stores a computer program, and the computer program is used to execute the method for processing live audio and video streams as provided by the embodiments of the present disclosure.
  • An embodiment of the present disclosure also provides a computer program, including: instructions, which when executed by a processor cause the processor to execute the method for processing live audio and video streams provided by the embodiment of the present disclosure.
  • Figure 1 is a schematic diagram of the processing process of a live audio and video stream according to related technologies
  • Figure 2 is a schematic diagram of the processing process of live audio and video streams according to one embodiment of the present disclosure
  • Figure 3 is a schematic flowchart of a method for processing live audio and video streams provided by an embodiment of the present disclosure
  • Figure 4 is a schematic flowchart of another method for processing live audio and video streams provided by an embodiment of the present disclosure
  • Figure 5 is a schematic diagram of a processing scenario for live audio and video streams according to an embodiment of the present disclosure
  • Figure 6 is a schematic diagram of a processing scenario for live audio and video streams according to another embodiment of the present disclosure.
  • Figure 7 is a schematic diagram of a processing scenario for live audio and video streams according to another embodiment of the present disclosure.
  • Figure 8 is a schematic flowchart of another method for processing live audio and video streams provided by an embodiment of the present disclosure.
  • Figure 9 is a schematic flowchart of another method for processing live audio and video streams provided by an embodiment of the present disclosure.
  • Figure 10 is a schematic diagram of a processing scenario for live audio and video streams according to another embodiment of the present disclosure.
  • Figure 11 is a schematic diagram of the processing process of live audio and video streams according to another embodiment of the present disclosure.
  • Figure 12 is a schematic structural diagram of a processing device for live audio and video streams provided by an embodiment of the present disclosure
  • Figure 13 is a schematic structural diagram of another device for processing live audio and video streams provided by an embodiment of the present disclosure.
  • FIG. 14 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • the term “include” and its variations are open-ended, ie, “including but not limited to.”
  • the term “based on” means “based at least in part on.”
  • the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; and the term “some embodiments” means “at least some embodiments”. Relevant definitions of other terms will be given in the description below.
  • client A on one side of the live broadcast sends a live broadcast connection request to client B on the other side of the live broadcast.
  • B receives the request, if it performs the permission operation, it sends a message allowing the live broadcast connection to A, and A receives the After receiving the message that the live broadcast connection is allowed, A and B join the live broadcast communication channel. Based on the communication channel, A and B both push their own audio and video streams to the remote end, so as to facilitate the combined playback process based on the audio and video streams.
  • the above refers to the live broadcast connection operation.
  • the invited party B After the invited party B performs the allow operation of the live broadcast connection, it has to go through a series of processes before it can see the screen of the inviting party A and hear the voice of the inviting party A.
  • the same inviting party A also needs to receive the invitation. It will take a long time after the inviter B's live broadcast connection permission message to see B's screen. As a result, there is a waiting delay in the live broadcast connection.
  • the present disclosure provides a processing method, device, equipment and medium for live audio and video streams to solve the problem of low picture rendering efficiency during live connection in related technologies.
  • the first client responds to the live broadcast connection operation with the second client, sends a live broadcast connection request for the second client, and then receives the second live broadcast connection request corresponding to the second client.
  • the second live audio and video stream is to start pulling the second live audio and video stream of the second client in advance, and in response to the received live broadcast connection message corresponding to the second client, the combined audio and video stream is played, wherein the combined audio and video stream is
  • the video stream is generated based on the confluence of the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client, and the live broadcast connection pass message is a pass message for the live broadcast connection request. This improves the screen rendering efficiency during live broadcast connections and avoids rendering waiting delays.
  • embodiments of the present disclosure provide a method for processing live audio and video streams.
  • this method as shown in Figure 2, the other party's live audio and video streams are pulled and one's own live audio is pushed.
  • Operations such as video streaming, Directly after A performs the live broadcast connection operation, all that needs to be done after receiving B's live broadcast connection pass message is to render the screen and trigger the merge. The time the user needs to wait is immediately shortened significantly. What needs to be done after performing the live broadcast connection confirmation operation The only thing that is done is rendering on the screen and triggering the merge playback process, and the time the user needs to wait is immediately shortened significantly.
  • the first client can be understood as the client of the inviting party of the live broadcast connection.
  • the first client side includes but is not limited to client devices, servers, etc.
  • the second client can be understood as the client of the invited party for live broadcast connection.
  • the second client side includes but is not limited to client devices, servers, etc.
  • Figure 3 is a schematic flowchart of a method for processing live audio and video streams provided by an embodiment of the present disclosure.
  • the method can be executed by a first client that is set up with a device for processing live audio and video streams.
  • the device can use software and/or Hardware implementation can generally be integrated in the electronic device where the first client is located.
  • the method includes:
  • Step 301 In response to the live broadcast connection operation with the second client, send a live broadcast connection request to the second client.
  • the live broadcast connection operation is usually a continuous wheather operation in a live broadcast scenario, that is, through the live broadcast connection operation, the first client and the second client can realize continuous wheather during the live broadcast.
  • the first client can be the invitation end, that is, the connection is sent.
  • the anchor end of the Mai invitation the second client can be the invited end, that is, the anchor end that receives the Mai invitation, and the live broadcast connection operation with the second client, that is, the live broadcast connection operation between the first client and the second client, can be through It can be triggered by the live broadcast connection control on the first client, or by performing a voice operation, or by performing a gesture operation, etc., which are not limited here.
  • a live broadcast connection request is sent to the second client.
  • a live broadcast connection request carrying the second client can be sent to the service server of the live broadcast connection.
  • the live broadcast connection request identified by the client is sent by the business server to the second client.
  • Step 302 Receive the second live audio and video stream corresponding to the second client.
  • the real-time reception operation of the second live audio and video stream of the second client is directly started.
  • a second live audio and video stream corresponding to the second client is received.
  • the first client subscribes to the second live audio and video stream of the second client.
  • the first server is a real time communication server (Real Time Communication, RTC)
  • the second client can subscribe to the real time communication server.
  • the server pushes the second live audio and video stream in real time, so that the first client can send a message to the preset real-time communication server.
  • Send an acquisition request for the second live audio and video stream where the acquisition request includes the client identifier of the second client, and further obtains the second live audio and video stream of the second client fed back by the real-time communication server, thus ensuring that The second live audio and video stream is pulled in time, further improving the live broadcast connection efficiency.
  • the acquisition granularity of the first client is to obtain the second live audio and video frame by frame.
  • the second live audio and video frame of the stream in order to reduce the memory pressure of the first client, the first client only saves the latest second live audio and video frame currently obtained. Every time a second live audio and video frame is acquired, the upcoming second live audio and video frame is The second live audio and video frame acquired in one frame is cleared.
  • the number of saved audio and video frames can also be determined based on the remaining memory of the first client. For example, the remaining memory of the first client is obtained, and the preset corresponding relationship between the remaining memory and the remaining memory can be queried. The number of audio and video frames corresponding to the memory is stored. Therefore, every time the first client obtains a second audio and video frame of the second client, it stores the corresponding second audio and video frame and performs counting processing. When the counting result reaches the preset audio and video frame, After storing the number of video frames, it is considered that the maximum capacity of the remaining memory has been reached. At this time, the historically acquired second video frame is deleted based on the currently acquired second video frame. Specifically, the first client deletes the second audio frame acquired next.
  • Second audio and video frames every time a new second audio and video frame is acquired, the first acquired second audio and video frame among the currently stored second audio and video frames is deleted, and the acquired new frame is saved. Second audio and video frames, therefore, the first client only saves the latest audio and video frame currently obtained and stores the number of second live audio and video frames. Every time a frame of the second live audio and video frame is obtained, the first one will be obtained in time. One frame of the second live audio and video frame is cleared. Therefore, by storing multiple frames of the second live audio and video stream of the second live audio and video stream, when an error occurs in the converging process, confluence retry can be performed.
  • Step 303 In response to the received live broadcast connection pass message corresponding to the second client, play the combined audio and video stream, wherein the combined audio and video stream is based on the second live audio and video stream corresponding to the second client and the first client
  • the corresponding first live broadcast audio and video streams are obtained by converging processing, wherein the live broadcast connection pass message is a pass message for the live broadcast connection request.
  • the live broadcast connection passed message of the second client fed back according to the live broadcast connection request is obtained, for example, the live broadcast connection passed message of the second client sent by the business server is obtained, the obtained second client's live broadcast connection passed message is directly read.
  • the second live audio and video frame corresponding to the live audio and video stream means that there is no need to pull the second live audio and video stream corresponding to the real-time acquisition operation, but based on the second live audio and video frame in the second live audio and video stream obtained in advance.
  • the video frame is combined and played according to the second live audio and video frame in the second live audio and video stream and the first live audio and video frame in the first live audio and video stream of the first client, that is, the playback is combined with the second live audio and video frame.
  • the second live audio and video frame in the stream and the first guest The combined audio and video stream corresponding to the first live audio and video frame in the client's first live audio and video stream greatly improves the picture rendering efficiency during live connection.
  • the combined audio and video streams may be fed back to the first client after reasonable processing by a preset second server.
  • a converging processing request is sent to a preset second server, where the converging processing request carries the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client. stream, the second server generates the combined audio and video stream and feeds it back to the first client for playback by the first client, thereby reducing the computing power requirements of the first client.
  • the combined audio and video stream is generated locally on the first client, thereby avoiding the need to communicate with the second server to obtain the corresponding combined audio and video stream, which is also possible when the network speed is poor. It does not affect the acquisition of combined audio and video streams.
  • the first client performs a confluence process on the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client to obtain the combined audio and video stream.
  • the business server responds to the second live broadcast connection cancellation message fed back according to the live broadcast connection request. If the live connection cancellation message of the second client is received, the push of the first live audio and video stream of the first client to the preset real-time communication server will be stopped to avoid the problem caused by the first client continuing to push the first live audio and video stream to the first server. Waste of network resources. In this embodiment, you can also stop acquiring the second live audio and video stream of the second client, and delete the acquired second live audio and video stream of the second client to reduce the impact on the first live audio and video stream. Client memory pressure.
  • the first client in order to facilitate the second client to pull the first live audio and video stream of the first client in advance, the first client begins to perform real-time communication to the default in response to the live broadcast connection operation.
  • the server pushes the first live audio and video stream of the first client.
  • the first client responds to the live broadcast connection operation with the second client, sends a live broadcast connection request for the second client, and then receives the live broadcast connection request corresponding to the second client.
  • the second live audio and video stream that is, the second live audio and video stream of the second client is pulled in advance, and the combined audio and video stream is played in response to the received live broadcast connection message corresponding to the second client, where the combined audio and video stream is played.
  • the audio and video streams are generated based on the confluence of the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client, and the live broadcast connection pass message is a pass message for the live broadcast connection request.
  • the combined playback process can be directly performed based on the pre-pulled live audio and video streams, saving the time of receiving the live broadcast connection.
  • the length of time for streaming after the live connection passes the message improves the screen rendering efficiency during the live connection and avoids rendering waiting delays.
  • the first client merges the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client to obtain the combined audio and video stream in different ways. , examples are as follows:
  • converging the second live audio and video stream of the second client and the first live audio and video stream of the first client includes:
  • Step 401 Determine the reception time of the live broadcast connection pass message.
  • the system time when the live broadcast connection is received through the message is determined, and the system time is determined as the reception time.
  • Step 402 Determine the second live audio and video frame corresponding to the reception time in the second live audio and video stream.
  • Step 403 Determine the first live audio and video frame corresponding to the reception time in the first live audio and video stream.
  • step 402 and step 403 are not limited, and they can also be executed at the same time.
  • the second live audio and video frame in the latest pulled second live audio and video stream at the current receiving time can be obtained, which is the second live audio and video frame corresponding to the receiving time. It is also possible to identify the push time of each audio and video frame in the second live audio and video stream, match the push time with the reception time, and determine that the successfully matched audio and video frame is the second audio and video frame.
  • the first live audio and video frame in the latest first live audio and video stream at the current reception time can be obtained, and the first live audio and video frame can be used as the first live audio and video frame corresponding to the reception time.
  • video frame or you can also identify the push moment of each first live audio and video frame in the first live audio and video stream, match the push moment with the reception time, and determine that the successfully matched audio and video frame is the first live audio and video frame.
  • Video frames or you can also identify the push moment of each first live audio and video frame in the first live audio and video stream, match the push moment with the reception time, and determine that the successfully matched audio and video frame is the first live audio and video frame.
  • Step 404 Taking the first live audio and video frame and the second live audio and video frame as the starting frame, perform confluence processing on the second live audio and video stream and the corresponding audio and video frames in the first live audio and video stream to obtain the combined audio and video frames.
  • Video streaming Taking the first live audio and video frame and the second live audio and video frame as the starting frame, perform confluence processing on the second live audio and video stream and the corresponding audio and video frames in the first live audio and video stream to obtain the combined audio and video frames.
  • Video streaming Taking the first live audio and video frame and the second live audio and video frame as the starting frame, perform confluence processing on the second live audio and video stream and the corresponding audio and video frames in the first live audio and video stream to obtain the combined audio and video frames.
  • the first live audio and video frame and the second audio and video frame are used as the starting frames, and based on the UI structure information of the first client, etc., the first live audio and video frame is merged.
  • the first video frame and the second video frame of the second live audio frame are rendered to obtain the merged video frame.
  • the merged video frame can be shown in Figure 6.
  • the first video of the first client is displayed on the left Frame, the right side shows the second video frame of the second client.
  • the first audio frame in the first live audio and video frame and the second audio frame in the second live audio frame are combined and played.
  • the first current audio and video frame in the first live audio and video stream, and the second live audio and video stream are determined.
  • the second current audio and video frame in the video stream merge the first current audio and video frame and the second current audio and video frame to obtain candidate merged audio and video frames, obtain the preset rendering parameters, and render the corresponding live broadcast connection according to the rendering parameters Display element.
  • the live broadcast connection display element can be the "PK progress bar" as shown in Figure 7, etc.
  • the rendering parameters include the display content, display mode, display position, display color, etc. of the corresponding display element.
  • the live connection display element is rendered in the candidate merged audio and video frame to obtain the target merged audio and video frame, and the merged audio and video stream composed of the continuous target merged audio and video frames is displayed, thus ensuring the merged display effect.
  • the second live audio and video stream corresponding to the real-time acquisition operation and the first live audio and video stream of the first client are merged, in order to further improve the screen display efficiency during the live broadcast, you can also When the live broadcast connection pass message is obtained, the most recently acquired second audio and video frame is read, and the current latest acquired second audio and video frame is continuously combined with the first audio and video frame in the first live audio and video stream. Merge.
  • fixed elements in the first live audio and video frame and the second live audio and video frame can also be determined.
  • the fixed elements in the first live audio and video frame and the second live video frame are determined.
  • the element type determines the fixed elements according to the element type, for example, the host's avatar information, for example, the live broadcast background information. Therefore, when merging the first live audio and video frame and the second live video frame, the first live audio frame of the previous frame is multiplexed.
  • the merged image area of the video frame and the second live broadcast fixed element is only the image area corresponding to the non-fixed element, which greatly improves the merging efficiency.
  • the processing method of live audio and video streams in the embodiment of the present disclosure directly performs the merge display processing after obtaining the live broadcast connection pass message, which improves the picture rendering efficiency during the live broadcast connection.
  • Figure 8 is a flow chart of a method for processing live audio and video streams according to an embodiment of the present disclosure. As shown in Figure 8, the method includes:
  • Step 801 In response to the received live broadcast connection request of the first client, obtain the first live audio and video stream corresponding to the first client.
  • the live broadcast connection request sent by the first client may be forwarded by the service server.
  • the first live broadcast audio and video stream of the first client is pulled. Since the first client already in advance Push the first live audio and video stream to the preset real-time communication server. Therefore, in one embodiment of the present disclosure, you can subscribe to the first live audio and video stream of the first client through the preset first server, such as the real-time communication server. .
  • the acquisition granularity of the second client is to acquire the first live audio and video frame in the first live audio and video stream frame by frame.
  • the second client only saves the latest first live audio and video frame currently obtained. Every time a first live audio and video frame is acquired, the previous first live audio and video frame is cleared.
  • the number of saved audio and video frames can be determined based on the remaining memory of the second client. For example, the remaining memory of the second client is obtained, and the number of audio and video frames stored corresponding to the remaining memory determined by querying the preset correspondence relationship is obtained. Therefore, the second client only saves the latest first live audio and video frame currently obtained and stores the number of first live audio and video frames. Every time a first live audio and video frame is obtained, the first frame in time will be obtained. The first live audio and video frame is cleared. Therefore, storing multiple frames of the first live audio and video frames allows for retrying the merge when an error occurs in the merge processing.
  • the second client also responds to the live broadcast connection request and starts to push the second live audio and video stream of the second client to the preset real-time communication server, so that the first client can pull it in advance. Get the second live audio and video stream of the second client.
  • Step 802 in response to the live broadcast connection confirmation operation, feedback the live broadcast connection pass message, and play the combined audio and video stream, where the combined audio and video stream is based on the first live audio and video stream corresponding to the first client and the second client.
  • the second live broadcast audio and video streams are obtained by merging processing, wherein the live broadcast connection confirmation operation is a confirmation connection operation for the live broadcast connection request, and the live broadcast connection pass message is a pass message for the live broadcast connection request.
  • the live broadcast connection operation in response to the live broadcast connection confirmation operation for the live broadcast connection request, may be triggered by the user's voice of the second client, may be triggered by the gesture track, or may be triggered by The corresponding connection control is triggered, etc.
  • the live broadcast connection pass message of the first client can be fed back to the business server, and then the combined audio and video stream can be played, thereby improving the efficiency of combined playback.
  • the combined audio and video stream is obtained by combining the first live audio and video stream corresponding to the first client and the second live audio and video stream corresponding to the second client.
  • the combined audio and video streams may be fed back to the second client after reasonable processing by the preset second server.
  • the confluence point is sent to the preset second server. processing request, wherein the converging processing request carries the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client, and the preset second server generates feedback after the converging audio and video stream To the second client for playback by the second client.
  • the combined audio and video stream is generated locally on the second client, so that even when the network of the second client is not good, the combined audio and video stream can be obtained.
  • the second client performs a confluence process on the second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client to obtain the combined audio and video stream.
  • the second client performs a live broadcast connection cancellation operation, that is, the second client performs a cancellation operation on the live broadcast connection request, such as triggering a "cancel" control, etc., then the live broadcast connection is fed back to the business server.
  • the second client responds to the received live broadcast connection request of the first client, obtains the first live audio and video stream corresponding to the first client, and then , in response to the live broadcast connection confirmation operation, feedback the live broadcast connection pass message, and play the combined audio and video stream, where the combined audio and video stream is based on the first live audio and video stream corresponding to the first client and the second live audio and video stream corresponding to the second client.
  • the live audio and video streams are obtained by merging the streams, wherein the live broadcast connection confirmation operation is a confirmation connection operation for the live broadcast connection request, and the live broadcast connection pass message is a pass message for the live broadcast connection request.
  • the live audio and video stream of the opposite client is obtained in advance, so that after obtaining the live broadcast connection permission operation, the first live audio and video stream corresponding to the first client and the second live broadcast corresponding to the second client are directly played.
  • the combined audio and video stream obtained by merging the audio and video streams improves the picture rendering efficiency during live connection and avoids waiting delays.
  • the second client locally performs a confluence process based on the first live audio and video stream corresponding to the first client and the second live audio and video stream corresponding to the second client.
  • the methods of merging the corresponding first live audio and video stream and the second live audio and video stream of the second client are different. Examples are as follows:
  • converging the corresponding first live audio and video stream and the second live audio and video stream of the second client includes:
  • Step 901 Determine the operation time of the live broadcast connection confirmation operation.
  • the system time at the time of the live broadcast connection confirmation operation is determined, and the system time is determined as the operation time.
  • Step 902 Determine the second live audio and video frame corresponding to the operation time in the second live audio and video stream.
  • Step 903 Determine the first live audio and video frame corresponding to the operation time in the first live audio and video stream.
  • step 902 and step 903 are not limited, and they can also be executed at the same time.
  • the first live audio and video frame in the latest pulled first live audio and video stream at the current operation time can be obtained, which is the first live audio and video frame corresponding to the operation time, It is also possible to identify the push moment of each audio and video frame in the first live audio and video stream, match the push moment with the operation time, and determine that the successfully matched audio and video frame is the first live audio and video frame.
  • the second live audio and video frame in the latest second live audio and video stream at the current operation time can be obtained, and the second live audio and video frame can be used as the second live audio and video frame corresponding to the operation time.
  • video frame or you can also identify the push moment of each second live audio and video frame in the second live audio and video stream, match the push moment with the operation time, and determine that the successfully matched audio and video frame is the second live audio and video frame.
  • Video frames or you can also identify the push moment of each second live audio and video frame in the second live audio and video stream, match the push moment with the operation time, and determine that the successfully matched audio and video frame is the second live audio and video frame.
  • Step 904 Taking the first live audio and video frame and the corresponding second live audio and video frame as the starting frame, perform a confluence process on the second live audio and video stream and the corresponding audio and video frame in the first live audio and video stream to obtain Combined audio and video streams.
  • the first live broadcast audio and video frame and the second live broadcast audio and video frame are used as the starting frames, and the video frames in the first live broadcast audio and video frame and the second live broadcast are merged according to the UI structure information of the first client, etc.
  • the video frame of the audio frame is merged to obtain the merged video frame.
  • the merged video frame can be shown in Figure 10.
  • the left side displays the second live video frame of the second client, and the right side displays the first client's second live video frame.
  • the first audio frame in the first live audio and video frame and the second audio frame in the second live audio frame are combined.
  • the first live audio and video frame of the next frame of the first live audio and video stream and the second live audio and video frame of the next live broadcast audio and video stream of the second live audio and video stream obtained in real time are read, and the above-mentioned converging operation is performed.
  • the first current audio and video frame in the first live audio and video stream and the second live audio and video stream are determined.
  • the second current audio and video frame in merge the first current audio and video frame and the second current audio and video frame to obtain candidate merged audio and video frames, obtain the preset rendering parameters, render and display the live broadcast connection display element according to the rendering parameters,
  • the live broadcast connection display element is an interface element displayed by the first client and the second client in a live broadcast connection (such as wheat connection, connection) scenario.
  • a live broadcast connection such as wheat connection, connection
  • the rendering parameters include the corresponding live broadcast connection.
  • the display content, display mode, display position, display color, etc. of the display element are then rendered.
  • the live connection display element is rendered in the candidate merged audio and video frames according to the rendering parameters to obtain the target merged audio and video frame, and the merged audio and video frame is obtained according to the target merged audio and video frame. Audio and video streaming.
  • the first live audio and video stream corresponding to the real-time acquisition operation and the second client when the second live audio and video streams are merged, in order to further improve the screen display efficiency during live broadcast, you can also read the most recently acquired first live audio and video frame when obtaining the live broadcast connection confirmation operation, and continue to add the current The latest acquired first live audio and video frame is merged with the second live audio and video frame in the second live audio and video stream.
  • fixed elements in the first live audio and video frame and the second live audio and video frame can also be determined, for example, determining the first live audio and video frame and the element type of each image element in the second live audio and video frame.
  • Fixed elements are determined according to the element type, such as live broadcast background information. Therefore, when merging the first live audio and video frame and the second live audio and video frame, multiplexing The merged image areas of the fixed elements of the first live audio and video frame and the second live audio of the previous frame only merge the image areas corresponding to the non-fixed elements, which greatly improves the merging efficiency.
  • the first client is A
  • the second client is B
  • a and B's relevant messages are interacted through the business server, and the live broadcast connections between A and B are transmitted in real time through RTC.
  • An example is as follows:
  • the RTC rendering channel of the live broadcast is closed, but starts to push A's first live audio and video frame to the RTC, and pulls B's second live audio and video stream.
  • the RTC rendering channel of the live broadcast is closed, but starts to push B's second live audio and video frame to the RTC, and pulls A's first live audio and video stream.
  • B performs the live broadcast connection permission confirmation operation, it can send a live broadcast connection pass message to A through the business server, etc.
  • the live RTC rendering channel is opened
  • the live broadcast connection channel is opened
  • the A side immediately Render the other party's video (including UI adjustment and adding the other party's surfaceview, etc.), play the other party's audio, and trigger the merge operation.
  • B's RTC rendering channel is opened, the live broadcast connection channel is opened, and the other party's video is immediately rendered on the B side (including UI adjustment and adding the other party's surfaceview, etc.), the other party's audio is played, and the merge operation is triggered. .
  • B If B performs a live broadcast connection cancellation operation, it can send a live broadcast connection cancellation message to A through the business server, etc. After A receives the live broadcast connection cancellation message, it clears the obtained second live audio and video stream of B and stops acquiring B. The second live audio and video stream. Similarly, B clears the acquired first live audio and video stream of A and stops acquiring A's first live audio and video stream.
  • Side B can determine whether the live broadcast is currently based on the number of users in the information list. For example, if the number of users in the list message is 2, it means that the live broadcast connection channels of A and B have been established. Among them, during the merge display, other live broadcast connection display elements (such as live broadcast progress bars, etc.) can be displayed on the merged video frame to increase the interest of the live broadcast.
  • live broadcast connection display elements such as live broadcast progress bars, etc.
  • the method for processing live audio and video streams in the embodiment of the present disclosure directly plays the first live audio and video stream corresponding to the first client and the second live broadcast corresponding to the second client after obtaining the live broadcast connection confirmation operation.
  • the combined audio and video stream obtained after merging the audio and video streams improves the picture rendering efficiency during live broadcast connection.
  • the present disclosure also proposes a processing device for live audio and video streams.
  • Figure 12 is a schematic structural diagram of a device for processing live audio and video streams provided by an embodiment of the present disclosure.
  • the device can be implemented by software and/or hardware, and can generally be integrated on the side of the electronic device where the first client is located. As shown in Figure 12, the device includes:
  • the sending module 1210 is configured to send a live broadcast connection request to the second client in response to the live broadcast connection operation with the second client;
  • the first receiving module 1220 is used to receive the second live audio and video stream corresponding to the second client;
  • the first combined playback module 1230 is configured to play the combined audio and video stream in response to the received live broadcast connection pass message corresponding to the second client, wherein the combined audio and video stream is based on the second live audio and video corresponding to the second client.
  • the stream is obtained by converging the first live audio and video stream corresponding to the first client, where the live connection pass message is a pass message for the live broadcast connection request.
  • the processing device for live audio and video streams provided by the embodiments of the present disclosure can execute the processing method of live audio and video streams provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method. Its implementation principles are similar. Here, No longer.
  • Figure 13 is a schematic structural diagram of a device for processing live audio and video streams provided by an embodiment of the present disclosure.
  • the device can be implemented by software and/or hardware, and can generally be integrated at the electronic equipment manufacturer where the second client is located. As shown in Figure 13, the device includes:
  • the second receiving module 1310 is configured to obtain the first live audio and video stream corresponding to the first client in response to the received live broadcast connection request of the first client;
  • Feedback module 1320 configured to respond to the live broadcast connection confirmation operation and feedback the live broadcast connection pass message
  • the second combined playback module 1330 is used to play the combined audio and video streams in response to the live broadcast connection confirmation operation, wherein, The merged audio and video stream is obtained by merging the first live audio and video stream corresponding to the first client and the second live audio and video stream corresponding to the second client, where the live broadcast connection confirmation operation is a confirmation of the live broadcast connection request.
  • the live broadcast connection pass message is a pass message for the live broadcast connection request.
  • the processing device for live audio and video streams provided by the embodiments of the present disclosure can execute the processing method of live audio and video streams provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method. Its implementation principles are similar. Here, No longer.
  • modules may be implemented as software components executing on one or more general-purpose processors, or may be implemented as hardware, such as programmable logic devices and/or application-specific integrated circuits, for performing certain functions.
  • these modules may be embodied in the form of a software product that may be stored in a non-volatile storage medium.
  • non-volatile storage media include enabling computer equipment (such as personal computers, servers, network equipment, mobile terminals, etc.) to execute the methods described in the embodiments of the present disclosure.
  • the above modules can also be implemented on a single device or distributed on multiple devices. The functionality of these modules can be combined with each other or further split into sub-modules.
  • the present disclosure also proposes a computer program product, which includes a computer program/instructions.
  • a computer program product which includes a computer program/instructions.
  • the computer program/instructions are executed by a processor, the method for processing live audio and video streams in the above embodiments is implemented.
  • a computer program is also provided.
  • the computer program includes instructions that, when executed by a processor, cause the processor to implement the method for processing live audio and video streams in the above embodiments.
  • FIG. 14 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • the electronic device 1400 in the embodiment of the present disclosure may include, but is not limited to, mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablets), PMPs (Portable Multimedia Players), vehicle-mounted terminals ( Mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers, etc.
  • the electronic device shown in FIG. 14 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
  • electronic device 1400 may include a processor (eg, central processing unit, graphics processor, etc.) 1401 that may be loaded into random access memory according to a program stored in read-only memory (ROM) 1402 or from memory 1408 (RAM) 1403 to perform various appropriate actions and processes.
  • ROM read-only memory
  • RAM memory 1408
  • the processor 1401, ROM 1402, and RAM 1403 are connected to each other through a bus 1404.
  • An input/output (I/O) interface 1405 is also connected to bus 1404.
  • the following devices may be connected to the I/O interface 1405: input devices 1406 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 1407 such as a computer; a memory 1408 including a magnetic tape, a hard disk, etc.; and a communication device 1409.
  • the communication device 1409 may allow the electronic device 1400 to communicate wirelessly or wiredly with other devices to exchange data.
  • FIG. 14 illustrates electronic device 1400 with various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 1409, or from memory 1408, or from ROM 1402.
  • the processor 1401 When the computer program is executed by the processor 1401, the above functions defined in the processing method of live audio and video streams according to the embodiment of the present disclosure are performed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmd read-only memory (EPROM or flash memory), fiber optics, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.
  • the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
  • Communications e.g., communications network
  • communications networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or developed in the future network of.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the computer-readable medium carries one or more programs.
  • the first client sends a message directed to the second client in response to the live broadcast connection operation with the second client.
  • the live broadcast connection request and then receiving the second live audio and video stream corresponding to the second client, that is, starting to pull the second live audio and video stream of the second client in advance, in response to the received live broadcast connection corresponding to the second client
  • the combined audio and video stream is played, wherein the combined audio and video stream is generated based on the combined second live audio and video stream corresponding to the second client and the first live audio and video stream corresponding to the first client.
  • the live broadcast The connection pass message is a pass message for the live broadcast connection request.
  • the time between sending the live broadcast connection request and receiving the live broadcast connection pass message can be fully utilized to obtain the live audio and video stream of the opposite terminal in advance, so that after the live broadcast connection pass message of the opposite terminal is obtained, the merge display processing can be directly performed. Improved screen rendering efficiency during live broadcast connections and avoided rendering waiting delays.
  • the electronic device may have computer program code for performing operations of the present disclosure written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, or a combination thereof. , also includes conventional procedural programming languages—such as "C" or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through Internet connection).
  • LAN local area network
  • WAN wide area network
  • Internet service provider such as an Internet service provider through Internet connection
  • each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two boxes represented one after another While they may actually be executed essentially in parallel, they may sometimes be executed in reverse order, depending on the functionality involved.
  • each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.
  • the units involved in the embodiments of the present disclosure can be implemented in software or hardware. Among them, the name of a unit does not constitute a limitation on the unit itself under certain circumstances.
  • FPGAs Field Programmable Gate Arrays
  • ASICs Application Specific Integrated Circuits
  • ASSPs Application Specific Standard Products
  • SOCs Systems on Chips
  • CPLD Complex Programmable Logical device
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM portable compact disk read-only memory
  • magnetic storage device or any suitable combination of the above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本公开实施例涉及一种直播音视频流的处理方法、装置、设备及介质，其中该方法包括：第一客户端响应于与第二客户端的直播连接操作，发送针对第二客户端的直播连接请求；接收与第二客户端对应的第二直播音视频流；响应于接收到的第二客户端对应的直播连接通过消息，播放合流音视频流，其中，合流音视频流是根据第二直播音视频流和第一直播音视频流进行合流处理得到的，其中，直播连接通过消息为针对直播连接请求的通过消息。

Description

直播音视频流的处理方法、装置、设备及介质
相关申请的交叉引用
本申请是以申请号为202210395011.9，申请日为2022年04月14日的中国申请为基础，并主张其优先权，该中国申请的公开内容在此作为整体引入本申请中。
技术领域
本公开涉及视频处理技术领域，尤其涉及一种直播音视频流的处理方法、装置、设备及介质。
背景技术
随着计算机技术的发展，直播逐渐兴起，为了满足用户的多种直播需求，直播的功能也愈发多样化。比如，在直播时，直播的一方客户端可以作为邀请方邀请另一个直播的客户端进行直播连接，当另一个直播客户端允许直播连接后，可以实现两个直播间的音视频通话，从而实现一种直播互动的效果。
发明内容
本公开提供了一种直播音视频流的处理方法、装置、设备及介质。
本公开实施例提供了一种直播音视频流的处理方法，所述方法应用在第一客户端，包括以下步骤：响应于与第二客户端的直播连接操作，发送针对所述第二客户端的直播连接请求；接收与所述第二客户端对应的第二直播音视频流；响应于接收到的所述第二客户端对应的直播连接通过消息，播放合流音视频流，其中，所述合流音视频流是根据所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流进行合流处理得到的，其中，所述直播连接通过消息为针对所述直播连接请求的通过消息。
本公开实施例提供了一种直播音视频流的处理方法，所述方法应用在第二客户端，包括以下步骤：响应于接收到的第一客户端的直播连接请求，获取所述第一客户端对应的第一直播音视频流；响应于直播连接确认操作，反馈直播连接通过消息，并播放合流音视频流，其中，所述合流音视频流是根据所述第一客户端对应的第一直播音视频流和所述第二客户端对应的第二直播音视频流进行合流处理得到的，其中，所述直播连接确认操作为针对所述直播连接请求的确认连接操作，所述直播连接通过消息为针对所述直播连接请求的 通过消息。
本公开实施例还提供了一种直播音视频流的处理装置，所述装置应用在第一客户端，包括：发送模块，用于响应于与第二客户端的直播连接操作，发送对所述第二客户端的直播连接请求；第一接收模块，用于接收与所述第二客户端对应的第二直播音视频流；第一合流播放模块，用于响应于接收到的所述第二客户端对应的直播连接通过消息，播放合流音视频流，其中，所述合流音视频流是根据所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流进行合流处理得到的，其中，所述直播连接通过消息为针对所述直播连接请求的通过消息。
本公开实施例还提供了一种直播音视频流的处理装置，所述装置应用在第二客户端，包括：第二接收模块，用于响应于接收到的第一客户端的直播连接请求，获取所述第一客户端对应的第一直播音视频流；反馈模块，用于响应于直播连接确认操作，反馈直播连接通过消息；第二合流播放模块，用于响应于直播连接确认操作，播放合流音视频流，其中，所述合流音视频流是根据所述第一客户端对应的第一直播音视频流和所述第二客户端对应的第二直播音视频流进行合流处理得到的，其中，所述直播连接确认操作为针对所述直播连接请求的确认连接操作，所述直播连接通过消息为针对所述直播连接请求的通过消息。
本公开实施例还提供了一种电子设备，所述电子设备包括：处理器；用于存储所述处理器可执行指令的存储器；所述处理器，用于从所述存储器中读取所述可执行指令，并执行所述指令以实现如本公开实施例提供的直播音视频流的处理方法。
本公开实施例还提供了一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序用于执行如本公开实施例提供的直播音视频流的处理方法。
本公开实施例还提供了一种计算机程序，包括：指令，所述指令当由处理器执行时使所述处理器执行本公开实施例提供的直播音视频流的处理方法。
附图说明
结合附图并参考以下具体实施方式，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。贯穿附图中，相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的，原件和元素不一定按照比例绘制。
图1为根据相关技术的一种直播音视频流的处理过程示意图；
图2为根据本公开一个实施例的直播音视频流的处理过程示意图；
图3为本公开实施例提供的一种直播音视频流的处理方法的流程示意图；
图4为本公开实施例提供的另一种直播音视频流的处理方法的流程示意图；
图5为根据本公开一个实施例的直播音视频流的处理场景示意图；
图6为根据本公开另一个实施例的直播音视频流的处理场景示意图；
图7为根据本公开另一个实施例的直播音视频流的处理场景示意图；
图8为本公开实施例提供的另一种直播音视频流的处理方法的流程示意图；
图9为本公开实施例提供的另一种直播音视频流的处理方法的流程示意图；
图10为根据本公开另一个实施例的直播音视频流的处理场景示意图；
图11为根据本公开另一个实施例的直播音视频流的处理过程示意图；
图12为本公开实施例提供的一种直播音视频流的处理装置的结构示意图；
图13为本公开实施例提供的另一种直播音视频流的处理装置的结构示意图；
图14为本公开实施例提供的一种电子设备的结构示意图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
应当理解，本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
相关技术中，由直播的一方客户端A向直播的另一方客户端B发送直播连接请求，B接收到该请求后，若是执行了允许操作，则向A发送允许直播连接的消息，A接收到允许直播连接的消息后，A和B加入直播的通信通道，基于该通信通道A和B均向远端推送自身的音视频流，以便于基于该音视频流进行合流播放处理。
上述指直播连接操作，被邀请方B执行直播连接的允许操作之后，还要经历一系列流程才能看到邀请方A的画面和听到A的声音，同样的邀请方A也要在接收到被邀请方B的直播连接的允许消息后一大段时间，才能看到B的画面。从而，导致直播的连接存在等待延迟。
为了解决上述技术问题或者至少部分地解决上述技术问题，本公开提供了一种直播音视频流的处理方法、装置、设备及介质，以解决相关技术中，直播连接时画面渲染效率低的问题。
本公开实施例提供的技术方案与相关技术相比具有如下优点：
本公开实施例提供的直播音视频流的处理方案，第一客户端响应于与第二客户端的直播连接操作，发送针对第二客户端的直播连接请求，进而，接收与第二客户端对应的第二直播音视频流，即预先开始拉取第二客户端的第二直播音视频流，响应于接收到的第二客户端对应的直播连接通过消息，进行合流音视频流的播放，其中，合流音视频流是根据第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流生成的，直播连接通过消息为针对直播连接请求的通过消息。由此，提升了直播连接时的画面渲染效率，避免了渲染等待延时。
为了解决上述提到的，如图1所示，邀请方客户端A在邀请客户端B建立直播连接时，在接收到B的直播连接通过的消息后，A和B才会拉取对方的直播音视频流、渲染上屏、触发合流等。这意味着用户需要等待这一系列的操作完成才可以看到对方的直播音视频流，这样将会导致直播连接的等待延迟，影响用户的直播体验的问题。
为了解决上述提到的问题，本公开实施例提供了一种直播音视频流的处理方法，在该方法中，如图2所示，将拉取对方的直播音视频流以及推送自己的直播音视频流等操作， 直接在A执行直播连接操作之后，那在接收到B的直播连接通过消息后要做的就只有渲染上屏和触发合流，用户需要等待的时间立刻大幅缩短，在执行直播连接确认操作后要做的就只有渲染上屏和触发合流播放处理，用户需要等待的时间立刻大幅缩短。
为了更全面的说明本公开实施例的直播音视频流的处理方法，下面分别集中在第一客户端和第二客户端进行说明，其中，第一客户端可以理解为直播连接的邀请方的客户端，该第一客户端侧包括但不限于客户端设备、服务器等。第二客户端可以理解为直播连接被邀请方的客户端，该第二客户端侧包括但不限于客户端设备、服务器等。
下面首先集中在第一客户端说明，具体说明如下：
图3为本公开实施例提供的一种直播音视频流的处理方法的流程示意图，该方法可以由设置直播音视频流的处理装置的第一客户端执行，其中该装置可以采用软件和/或硬件实现，一般可集成在第一客户端所在的电子设备中。如图3所示，该方法包括：
步骤301，响应于与第二客户端的直播连接操作，发送对第二客户端的直播连接请求。
其中，直播连接操作通常为直播场景下的连麦操作、即通过该直播连接操作第一客户端和第二客户端可以实现直播时的连麦，第一客户端可以为邀请端、即发出连麦邀请的主播端，第二客户端可以为受邀端、即接收连麦邀请的主播端，与第二客户端的直播连接操作即第一客户端与第二客户端的直播连接操作，可以是通过第一客户端上的直播连接控件触发的，也可以是通过执行语音操作触发的，也可以是执行手势操作触发的等，在此不作限制。
在本实施例中，响应于与第二客户端的直播连接操作，发送对第二客户端的直播连接请求，其中，在一些可能的实施例中，可以向直播连接的业务服务器发送携带第二客户端的客户端标识的直播连接请求，由业务服务器将该直播连接请求发送给第二客户端。
步骤302，接收与第二客户端对应的第二直播音视频流。
在本实施例中，为了提高直播连接时的画面渲染效率，是在发送对第二客户端的直播连接请求后，直接开始执行对第二客户端的第二直播音视频流的实时接收操作。
具体地，在一些实施例中，响应于发送出对第二客户端的直播连接请求，接收与第二客户端对应的第二直播音视频流。
在本实施例中，第一客户端订阅第二客户端的第二直播音视频流，其中，若是第一服务器端为实时通信服务器(Real Time Communication，RTC)，则第二客户端可以向实时通信服务器实时推送第二直播音视频流，从而，第一客户端可以向预设的实时通信服务器发 送第二直播音视频流的获取请求，其中，获取请求中包括第二客户端的客户端标识，进而，获取实时通信服务器反馈的第二客户端的第二直播音视频流，由此，保证了可以及时的拉取到第二直播音视频流，进一步提高了直播连接效率。
需要说明的是，若是基于RTC服务器获取第二直播音视频流，则由于RTC实时传输第二客户端的第二直播音视频流，因此，第一客户端的获取粒度是逐帧获取第二直播音视频流的第二直播音视频帧，为了减轻第一客户端的内存压力，第一客户端仅仅保存当前获取到的最新的第二直播音视频帧，每获取一帧第二直播音视频帧，即将上一帧获取到的第二直播音视频帧进行清除。
当然，在一些可能的实施例中，也可以根据第一客户端的内存剩余，确定保存的音视频帧的数量，比如，获取第一客户端的剩余内存，查询预设对应关系确定的该与该剩余内存对应的音视频帧存储数量，从而，第一客户端每获取一帧第二客户端的第二音视频帧后，存储对应的第二音视频帧并进行计数处理，当计数结果达到预设音视频帧存储数量后，认为已经达到剩余内存的最大容量，此时，根据当前获取的第二视频帧删除历史获取的第二视频帧，具体而言，第一客户端对接下来获取的第二音视频帧而言，每获取一帧新的第二音视频帧，即删除当前存储的第二音视频帧中最先获取的一帧第二音视频帧，并保存该获取的一帧新的第二音视频帧，由此，第一客户端仅仅保存当前获取到的最新的音视频帧存储数量个第二直播音视频帧，每获取一帧第二直播音视频帧，即将获取时间最靠前的一帧第二直播音视频帧进行清除。由此，存储第二直播音视频流的多帧第二直播音视频帧可以在合流处理出现错误时，进行合流重试等。
步骤303，响应于接收到的第二客户端对应的直播连接通过消息，播放合流音视频流，其中，合流音视频流是根据第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流处理得到的，其中，直播连接通过消息为针对直播连接请求的通过消息。
在本实施例中，若是获取到根据直播连接请求反馈的第二客户端的直播连接通过消息，比如，获取到业务服务器发送的第二客户端的直播连接通过消息，则直接读取获取到的第二直播音视频流对应的第二直播音视频帧，即无需再去拉取实时获取操作对应的第二直播音视频流，而是根据预先获取到的第二直播音视频流中的第二直播音视频帧，根据第二直播音视频流中的第二直播音视频帧和第一客户端的第一直播音视频流中的第一直播音视频帧进行合流播放处理，即播放与第二直播音视频流中的第二直播音视频帧和第一客 户端的第一直播音视频流中的第一直播音视频帧对应的合流音视频流，大大提升了直播连接时的画面渲染效率。
其中，在一些可能的实施例中，合流音视频流可以是预设的第二服务端进行合理处理后，反馈给第一客户端的。
在本实施例中，向预设的第二服务端发送合流处理请求，其中，合流处理请求中携带第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流，第二服务端生成合流音视频流后反馈给第一客户端，以供第一客户端进行播放，由此，降低了对第一客户端的算力需求。
在另一些可能的实施例中，合流音视频流是在第一客户端的本地生成的，从而，避免了向第二服务端进行通信以获取对应的合流音视频流，在网速不佳时也不影响合流音视频流的获取。在本实施例中，第一客户端对第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流处理，以得到合流音视频流。
在本公开的一个实施例中，若是第二客户端执行了直播连接取消操作，则响应于根据直播连接请求反馈的第二客户端的直播连接取消消息，比如，业务服务器根据直播连接请求反馈的第二客户端的直播连接取消消息，则停止执行向预设的实时通信服务器推送第一客户端的第一直播音视频流，避免第一客户端持续向第一服务端推送第一直播音视频流导致的网络资源的浪费，在本实施例中，还可以停止获取第二客户端的第二直播音视频流，并对已经获取的第二客户端的第二直播音视频流进行删除等，以降低对第一客户端的内存的压力。
其中，在本公开的一个实施例中，为了便于第二客户端可以预先拉取第一客户端的第一直播音视频流，第一客户端响应于直播连接操作，开始执行向预设的实时通信服务器推送第一客户端的第一直播音视频流。
综上，本公开实施例的直播音视频流的处理方法，第一客户端响应于与第二客户端的直播连接操作，发送针对第二客户端的直播连接请求，进而，接收与第二客户端对应的第二直播音视频流，即预先拉取第二客户端的第二直播音视频流，响应于接收到的第二客户端对应的直播连接通过消息，进行合流音视频流的播放，其中，合流音视频流是根据第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流生成的，直播连接通过消息为针对直播连接请求的通过消息。由此，在接收到对侧终端的直播连接通过消息后，可以根据预先拉取的直播音视频流直接进行合流播放处理，节省了在接收到 直播连接通过消息之后再行拉流的时长，提升了直播连接时的画面渲染效率，避免了渲染等待延时。
在实际执行过程中，若是合流音视频流是在第一客户端的本地生成的。在不同的应用场景中，第一客户端对第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流处理，以得到合流音视频流的方式不同，示例如下：
在本公开的一个实施例中，如图4所示，对第二客户端的第二直播音视频流和第一客户端的第一直播音视频流进行合流处理，包括：
步骤401，确定直播连接通过消息的接收时刻。
在本实施例中，确定直播连接通过消息接收到时的系统时间，确定该系统时间为接收时刻。
步骤402，确定第二直播音视频流中与接收时刻对应的第二直播音视频帧。
步骤403，确定第一直播音视频流中与接收时刻对应的第一直播音视频帧。
这里，步骤402和步骤403的执行顺序不受限制，也可以同时执行。
在本实施例中，为了实现实时合流效果，可以获取当前接收时刻下最新拉取的第二直播音视频流中的第二直播音视频帧，为与接收时刻对应的第二直播音视频帧，也可以识别第二直播音视频流中每个音视频帧的推流时刻，将该推流时刻与接收时刻匹配，确定匹配成功的音视频帧为第二音视频帧。
同样的，在本实施例中，可以获取当前接收时刻下最新的第一直播音视频流中的第一直播音视频帧，将该第一直播音视频帧作为与接收时刻对应的第一直播音视频帧，或者，也可以识别第一直播音视频流中每个第一直播音视频帧的推流时刻，将该推流时刻与接收时刻匹配，确定匹配成功的音视频帧为第一直播音视频帧。
步骤404，以第一直播音视频帧和第二直播音视频帧为起始帧，对第二直播音视频流和第一直播音视频流中的对应音视频帧执行合流处理，以得到合流音视频流。
在本实施例中，如图5所示，以第一直播音视频帧和第二音视频帧为起始帧，根据第一客户端的UI结构信息等，合并该第一直播音视频帧中的第一视频帧和第二直播音频帧的第二视频帧，渲染得到合并后的视频帧，该合并后的合流视频帧可如图6所示，左侧显示的是第一客户端的第一视频帧，右侧显示的是第二客户端的第二视频帧。同时，对第一直播音视频帧中的第一音频帧和第二直播音频帧的第二音频帧进行合流播放。
进而，读取实时获取的第二直播音视频流的下一帧第二直播音视频帧，和第一直播音频流的下一帧第一直播音视频帧，进行合流视频帧组成的合流视频流的合流处理。
其中，在合流处理时，还可以叠加直播连接场景下的其他指标连接显示元素，比如，在进行合流处理时，确定第一直播音视频流中的第一当前音视频帧，和第二直播音视频流中的第二当前音视频帧，合并第一当前音视频帧和第二当前音视频帧，以获取候选合并音视频帧，获取预设的渲染参数，根据该渲染参数渲染对应的直播连接显示元素，该直播连接显示元素可以为如图7所示的“PK进度条”等，渲染参数包括对应的显示元素的显示内容、显示方式、显示位置、显示颜色等，进而，根据渲染参数在候选合并音视频帧中渲染直播连接显示元素，以获取目标合并音视频帧，显示连续的目标合并音视频帧组成的合流音视频流，从而，保证了合流的显示效果。
在本公开的另一个实施例中，对实时获取操作对应的第二直播音视频流和第一客户端的第一直播音视频流进行合流处理时，为了进一步提升直播时候的画面显示效率，还可以在获取到直播连接通过消息时，读取最近一次获取到的第二音视频帧，持续将当前的最新一次获取到的第二音视频帧与第一直播音视频流中的第一音视频帧进行合并。
其中，在合并视频帧时，还可以确定第一直播音视频帧和第二直播音视频帧中的固定元素，比如，确定第一直播音视频帧和第二直播视频帧中每个图像元素的元素类型，根据元素类型确定固定元素，比如，主播的头像信息、比如，直播背景信息，因此，在合并第一直播音视频帧和第二直播视频帧时，复用上一帧第一直播音视频帧和第二直播固定元素的合并图像区域，仅仅对非固定元素对应的图像区域，大大提升了合并效率。
综上，本公开实施例的直播音视频流的处理方法，在获取到直播连接通过消息后，直接进行合流显示处理，提升了直播连接时的画面渲染效率。
下面其次集中在第二客户端说明本公开实施例的直播音视频流的处理方法。
图8是根据本公开一个实施例的直播音视频流的处理方法的流程图，如图8所示，该方法包括：
步骤801，响应于接收到的第一客户端的直播连接请求，获取第一客户端对应的第一直播音视频流。
其中，第一客户端发送的直播连接请求可以是业务服务器转发的，在本实施例中，获取到直播连接请求后，即拉取第一客户端的第一直播音视频流，由于第一客户端已经预先 向预设的实时通信服务器推送第一直播音视频流，因此，在本公开的一个实施例中，可以通过预设的第一服务端比如实时通信服务器订阅第一客户端的第一直播音视频流。
需要说明的是，由于实时传输第一客户端的第一直播音视频流，因此，第二客户端的获取粒度是逐帧获取第一直播音视频流中的第一直播音视频帧，为了减轻第二客户端的内存压力，第二客户端仅仅保存当前获取到的最新的第一直播音视频帧，每获取一帧第一直播音视频帧，即将上一帧第一直播音视频帧进行清除。
当然，可以根据第二客户端的内存剩余，确定保存的音视频帧的数量，比如，获取第二客户端的剩余内存，查询预设对应关系确定的该与该剩余内存对应的音视频帧存储数量，从而，第二客户端仅仅保存当前获取到的最新的第一直播音视频帧存储数量个第一直播音视频帧，每获取一帧第一直播音视频帧，即将获取时间最靠前的一帧第一直播音视频帧进行清除。由此，存储多帧第一直播音视频帧可以在合流处理出现错误时，进行合流重试等。
在本实施例中，同样的，第二客户端还响应于直播连接请求，开始执行向预设的实时通信服务器推送第二客户端的第二直播音视频流，以便于第一客户端能够预先拉取第二客户端的第二直播音视频流。
步骤802，响应于直播连接确认操作，反馈直播连接通过消息，并播放合流音视频流，其中，合流音视频流是根据第一客户端对应的第一直播音视频流和第二客户端对应的第二直播音视频流进行合流处理得到的，其中，直播连接确认操作为针对直播连接请求的确认连接操作，直播连接通过消息为针对直播连接请求的通过消息。
在本公开的一个实施例中，响应于对直播连接请求的直播连接确认操作，该直播连接操作，可以为第二客户端的用户语音触发的、可以为通过手势轨迹触发的、也可以为通过触发对应的连接控件触发的等。
响应于直播连接请求的直播连接确认操作，可以向业务服务器反馈第一客户端的直播连接通过消息，进而，播放合流音视频流，从而，提升了合流播放的效率。
其中，合流音视频流是根据第一客户端对应的第一直播音视频流和第二客户端对应的第二直播音视频流进行合流处理得到的。
其中，在一些可能的实施例中，合流音视频流可以是预设的第二服务端进行合理处理后，反馈给第二客户端的。
在本实施例中，为了降低对第二客户端的算力消耗，向预设的第二服务端发送合流处 理请求，其中，合流处理请求中携带第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流，预设的第二服务端生成合流音视频流后反馈给第二客户端，以供第二客户端进行播放。
在另一些可能的实施例中，合流音视频流是在第二客户端的本地生成的，从而，即使在第二客户端的网络不佳时，也可以获取合流音视频流。在本实施例中，第二客户端对第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流处理，以得到合流音视频流。在本公开的一个实施例中，若是第二客户端执行直播连接取消操作，即第二客户端执行对直播连接请求的取消操作，比如，触发“取消”控件等，则向业务服务器反馈直播连接取消消息，以便于第一客户端可以获取到该直播连接取消消息，停止执行向第一服务端推送第二客户端的第二直播音视频流，以避免持续推送导致的网络资源的浪费，在本实施例中，还可以停止获取第一客户端的第一直播音视频流，并对已经获取的第一客户端的第一直播音视频流进行删除等。
综上，本公开实施例的直播音视频流的处理方法，第二客户端响应于接收到的第一客户端的直播连接请求，获取所述第一客户端对应的第一直播音视频流，进而，响应于直播连接确认操作，反馈直播连接通过消息，并播放合流音视频流，其中，合流音视频流是根据第一客户端对应的第一直播音视频流和第二客户端对应的第二直播音视频流进行合流处理得到的，其中，直播连接确认操作为针对直播连接请求的确认连接操作，直播连接通过消息为针对所述直播连接请求的通过消息。由此，提前获取对侧的客户端的直播音视频流，从而在获取到直播连接允许操作后，直接播放根据第一客户端对应的第一直播音视频流和第二客户端对应的第二直播音视频流进行合流处理得到的合流音视频流，提升了直播连接时的画面渲染效率，避免了等待延时。
在实际执行过程中，在第二客户端本地根据第一客户端对应的第一直播音视频流和第二客户端对应的第二直播音视频流进行合流处理之，在不同的应用场景中，对对应的第一直播音视频流和第二客户端的第二直播音视频流进行合流处理的方式不同，示例如下：
在本公开的一个实施例中，如图9所示，对对应的第一直播音视频流和第二客户端的第二直播音视频流进行合流处理，包括：
步骤901，确定直播连接确认操作的操作时刻。
在本实施例中，确定直播连接确认操作时的系统时间，确定该系统时间为操作时刻。
步骤902，确定第二直播音视频流中与操作时刻对应的第二直播音视频帧。
步骤903，确定第一直播音视频流中与操作时刻对应的第一直播音视频帧。
这里，步骤902和步骤903的执行顺序不受限制，也可以同时执行。
在本实施例中，为了实现实时合流效果，可以获取当前操作时刻下最新拉取的第一直播音视频流中的第一直播音视频帧，为与操作时刻对应的第一直播音视频帧，也可以识别第一直播音视频流中每个音视频帧的推流时刻，将该推流时刻与操作时刻匹配，确定匹配成功的音视频帧为第一直播音视频帧。
同样的，在本实施例中，可以获取当前操作时刻下最新的第二直播音视频流中的第二直播音视频帧，将该第二直播音视频帧作为与操作时刻对应的第二直播音视频帧，或者，也可以识别第二直播音视频流中每个第二直播音视频帧的推流时刻，将该推流时刻与操作时刻匹配，确定匹配成功的音视频帧为第二直播音视频帧。
步骤904，以第一直播音视频帧和对应的第二直播音视频帧为起始帧，对第二直播音视频流和第一直播音视频流中的对应音视频帧执行合流处理，以得到合流音视频流。
在本实施例中，以第一直播音视频帧和第二音视频帧为起始帧，根据第一客户端的UI结构信息等，合并该第一直播音视频帧中的视频帧和第二直播音频帧的视频帧，得到合并后的视频帧，该合并后的视频帧可如图10所示，左侧显示的是第二客户端的第二直播视频帧，右侧显示的是第一客户端的第一直播视频帧。同时，对第一直播音视频帧中的第一音频帧和第二直播音频帧的第二音频帧进行合流。
进而，读取实时获取的第一直播音视频流的下一帧第一直播音视频帧，和第二直播音频流的下一帧第二直播音视频帧，进行上述合流操作。
其中，在合流处理时，还可以叠加直播连接场景下的其他显示元素，比如，在进行合流处理时，确定第一直播音视频流中的第一当前音视频帧，和第二直播音视频流中的第二当前音视频帧，合并第一当前音视频帧和第二当前音视频帧，以获取候选合并音视频帧，获取预设的渲染参数，根据该渲染参数渲染显示直播连接显示元素，该直播连接显示元素是第一客户端和第二客户端在直播连接(如连麦、连线)场景下显示的界面元素，例如可以为“PK进度条”等，渲染参数包括对应的直播连接显示元素的显示内容、显示方式、显示位置、显示颜色等，进而，根据渲染参数在候选合并音视频帧中渲染直播连接显示元素，以获取目标合并音视频帧，根据目标合并音视频帧得到合流音视频流。
在本公开的另一个实施例中，对实时获取操作对应的第一直播音视频流和第二客户端 的第二直播音视频流进行合流处理时，为了进一步提升直播时候的画面显示效率，还可以在获取到直播连接确认操作时，读取最近一次获取到的第一直播音视频帧，持续将当前的最新一次获取到的第一直播音视频帧与第二直播音视频流中的第二直播音视频帧进行合并。
其中，在合并第一直播音视频帧和第二直播音视频帧时，还可以确定第一直播音视频帧和第二直播音音视频帧中的固定元素，比如，确定第一直播音视频帧和第二直播音视频帧中每个图像元素的元素类型，根据元素类型确定固定元素，比如，直播背景信息，因此，在合并第一直播音视频帧和第二直播音视频帧时，复用上一帧第一直播音视频帧和第二直播音固定元素的合并图像区域，仅仅对非固定元素对应的图像区域进行合并处理，大大提升了合并效率。
为了更全面的说明本公开实施例的直播音视频流的处理方法，下面结合具体的应用场景进行举例，其中，在该场景下，第一客户端为A，第二客户端为B，A和B的有关消息通过业务服务器交互，A和B的直播连接通过RTC实时传输，举例说明如下：
如图11所示，当A向业务服务器发送直播连接请求后，直播的RTC渲染通道关闭，但是开始向RTC推送A的第一直播音视频帧，并拉取B的第二直播音视频流。
B在获取到A的直播连接请求后，直播的RTC渲染通道关闭，但是开始向RTC推送B的第二直播音视频帧，并拉取A的第一直播音视频流。
若是B执行直播连接允确认操作，则可以通过业务服务器等向A发送直播连接通过消息，A收到该直播连接通过消息后，则直播的RTC渲染通道开启，直播连接通道开启，在A侧立刻渲染对方的视频(包括UI调整以及增加对方的surfaceview等)，播放对方的音频，并触发合流操作。
B在执行直播连接确认操作后，B的RTC渲染通道开启，直播连接通道开启，在B侧立刻渲染对方的视频(包括UI调整以及增加对方的surfaceview等)，播放对方的音频，并触发合流操作。
若是B执行直播连接取消操作，则可以通过业务服务器等向A发送直播连接取消消息，A收到该直播连接取消消息后，则清空获取到的B的第二直播音视频流，并停止获取B的第二直播音视频流。同样的，B清空获取到的A的第一直播音视频流，并停止获取A的第一直播音视频流。
其中展开开启直播连接通道，B侧可以根据信息列表的用户数量判断是否当前正在直 播连接，比如，若是list message里面的用户数量为2，则表示A和B的直播连接通道已经建立。其中，在合流显示时，可以在合流视频帧上显示其他直播连接显示元素(比如直播进度条等)，以增加直播趣味性。
综上，本公开实施例的直播音视频流的处理方法，在获取到直播连接确认操作后，直接播放根据第一客户端对应的第一直播音视频流和第二客户端对应的第二直播音视频流合并处理后得到的合流音视频流，提升了直播连接时的画面渲染效率。
为了实现上述实施例，本公开还提出了一种直播音视频流的处理装置。
图12为本公开实施例提供的一种直播音视频流的处理装置的结构示意图，该装置可由软件和/或硬件实现，一般可集成在第一客户端所在的电子设备侧。如图12所示，该装置包括：
发送模块1210，用于响应于与第二客户端的直播连接操作，发送对第二客户端的直播连接请求；
第一接收模块1220，用于接收与第二客户端对应的第二直播音视频流；
第一合流播放模块1230，用于响应于接收到的第二客户端对应的直播连接通过消息，播放合流音视频流，其中，合流音视频流是根据第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流处理得到的，其中，直播连接通过消息为针对直播连接请求的通过消息。
本公开实施例所提供的直播音视频流的处理装置可执行本公开任意实施例所提供的直播音视频流的处理方法，具备执行方法相应的功能模块和有益效果，其实现原理类似，在此不再赘述。
图13为本公开实施例提供的一种直播音视频流的处理装置的结构示意图，该装置可由软件和/或硬件实现，一般可集成在第二客户端所在的电子设备商。如图13所示，该装置包括：
第二接收模块1310，用于响应于接收到的第一客户端的直播连接请求，获取第一客户端对应的第一直播音视频流；
反馈模块1320，用于响应于直播连接确认操作，反馈直播连接通过消息；
第二合流播放模块1330，用于响应于直播连接确认操作，播放合流音视频流，其中， 合流音视频流是根据第一客户端对应的第一直播音视频流和第二客户端对应的第二直播音视频流进行合流处理得到的，其中，直播连接确认操作为针对直播连接请求的确认连接操作，直播连接通过消息为针对直播连接请求的通过消息。
本公开实施例所提供的直播音视频流的处理装置可执行本公开任意实施例所提供的直播音视频流的处理方法，具备执行方法相应的功能模块和有益效果，其实现原理类似，在此不再赘述。
上述模块可以被实现为在一个或多个通用处理器上执行的软件组件，也可以被实现为诸如用于执行某些功能的硬件，诸如可编程逻辑器件和/或专用集成电路。在一些实施例中，这些模块可以体现为软件产品的形式，该软件产品可以存储在非易失性存储介质中。这些非易失性存储介质中包括使得计算机设备(例如个人计算机、服务器、网络设备、移动终端等)执行本公开实施例中描述的方法。在一些实施例中，上述模块还可以在单个设备上实现，也可以分布在多个设备上。这些模块的功能可以相互合并，也可以进一步拆分为多个子模块。
为了实现上述实施例，本公开还提出一种计算机程序产品，包括计算机程序/指令，该计算机程序/指令被处理器执行时实现上述实施例中的直播音视频流的处理方法。
本申请的一些实施例中，还提供了一种计算机程序，该计算机程序包括指令，所述指令当由处理器执行时使所述处理器实现上述实施例中的直播音视频流的处理方法。
图14为本公开实施例提供的一种电子设备的结构示意图。
下面具体参考图14，其示出了适于用来实现本公开实施例中的电子设备1400的结构示意图。本公开实施例中的电子设备1400可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图14示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图14所示，电子设备1400可以包括处理器(例如中央处理器、图形处理器等)1401，其可以根据存储在只读存储器(ROM)1402中的程序或者从存储器1408加载到随机访问存储器(RAM)1403中的程序而执行各种适当的动作和处理。在RAM 1403中，还存储 有电子设备1400操作所需的各种程序和数据。处理器1401、ROM 1402以及RAM 1403通过总线1404彼此相连。输入/输出(I/O)接口1405也连接至总线1404。
通常，以下装置可以连接至I/O接口1405：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置1406；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置1407；包括例如磁带、硬盘等的存储器1408；以及通信装置1409。通信装置1409可以允许电子设备1400与其他设备进行无线或有线通信以交换数据。虽然图14示出了具有各种装置的电子设备1400，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置1409从网络上被下载和安装，或者从存储器1408被安装，或者从ROM 1402被安装。在该计算机程序被处理器1401执行时，执行本公开实施例的直播音视频流的处理方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得：第一客户端响应于与第二客户端的直播连接操作，发送针对第二客户端的直播连接请求，进而，接收与第二客户端对应的第二直播音视频流，即预先开始拉取第二客户端的第二直播音视频流，响应于接收到的第二客户端对应的直播连接通过消息，进行合流音视频流的播放，其中，合流音视频流是根据第二客户端对应的第二直播音视频流和第一客户端对应的第一直播音视频流进行合流生成的，直播连接通过消息为针对直播连接请求的通过消息。由此，可以充分利用发送直播连接请求至接收直播连接通过消息的时间，提前获取对侧终端的直播音视频流，从而在获取到对侧终端的直播连接通过消息后，直接进行合流显示处理，提升了直播连接时的画面渲染效率，避免了渲染等待延时。
电子设备可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实 际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上系统(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (18)

  1. 一种直播音视频流的处理方法，所述方法应用在第一客户端，包括以下步骤：
    响应于与第二客户端的直播连接操作，发送针对所述第二客户端的直播连接请求；
    接收与所述第二客户端对应的第二直播音视频流；
    响应于接收到的所述第二客户端对应的直播连接通过消息，播放合流音视频流，其中，所述合流音视频流是根据所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流进行合流处理得到的，其中，所述直播连接通过消息为针对所述直播连接请求的通过消息。
  2. 如权利要求1所述的方法，其中，所述方法还包括：向第一服务端发送音视频流获取请求，其中，所述音视频流获取请求中包括所述第二客户端的客户端标识；
    所述接收与所述第二客户端对应的第二直播音视频流，包括：接收所述第一服务端反馈的所述第二客户端的第二直播音视频流。
  3. 如权利要求1或2所述的方法，其中，在所述播放合流音视频流之前，包括：
    对所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流进行合流处理，以得到合流音视频流；或者，
    向第二服务端发送合流处理请求，其中，所述合流处理请求中携带所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流，及接收所述第二服务端反馈的合流音视频流。
  4. 如权利要求3所述的方法，其中，所述对所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流进行合流处理，以得到合流音视频流，包括：
    确定所述直播连接通过消息的接收时刻；
    确定所述第一直播音视频流中与所述接收时刻对应的第一直播音视频帧以及所述第二直播音视频流中与所述接收时刻对应的第二直播音视频帧；
    以所述第一直播音视频帧和所述第二直播音视频帧为起始帧，对所述第二直播音视频流和所述第一直播音视频流中的对应音视频帧执行合流处理，以得到合流音视频流。
  5. 如权利要求4所述的方法，其中，所述对所述第二直播音视频流和所述第一直播音视频流中的对应音视频帧执行合流处理，以得到合流音视频流，包括：
    在所述合流处理时，确定所述第一直播音视频流中的第一当前音视频帧，和所述第二 直播音视频流中的第二当前音视频帧；
    合并所述第一当前音视频帧和所述第二当前音视频帧，以获取候选合并音视频帧；
    获取预设的渲染参数；
    根据所述渲染参数在所述候选合并音视频帧中渲染直播连接显示元素，以获取目标合并音视频帧；
    根据所述目标合并音视频帧得到所述合流音视频流。
  6. 如权利要求1-5任一所述的方法，还包括：
    若未接收到所述直播连接通过消息，则删除接收到的所述第二客户端的第二直播音视频流。
  7. 如权利要求2所述的方法，还包括：
    响应于接收到所述第二客户端对应的连接取消消息，停止执行向第一服务端推送所述第一客户端的第一直播音视频流。
  8. 一种直播音视频流的处理方法，所述方法应用在第二客户端，包括以下步骤：
    响应于第一客户端的直播连接请求，接收所述第一客户端对应的第一直播音视频流；
    响应于直播连接确认操作，反馈直播连接通过消息，并播放合流音视频流，其中，所述合流音视频流是根据所述第一客户端对应的第一直播音视频流和所述第二客户端对应的第二直播音视频流进行合流处理得到的，其中，所述直播连接确认操作为针对所述直播连接请求的确认操作，所述直播连接通过消息为针对所述直播连接请求的通过消息。
  9. 如权利要求8所述的方法，还包括：
    响应于所述直播连接请求，向第一服务端推送所述第二客户端的第二直播音视频流。
  10. 如权利要求8或9所述的方法，其中，在所述播放合流音视频流之前，包括：
    对所述第一客户端对应的第一直播音视频流和所述第二客户端对应的第二直播音视频流进行合流处理，以得到合流音视频流；或者，
    向第二服务端发送合流处理请求，其中，所述合流处理请求中携带所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流；及接收所述第二服务端反馈的合流音视频流。
  11. 如权利要求10所述的方法，其中，所述对所述第一客户端对应的第一直播音视频流和所述第二客户端对应的第二直播音视频流进行合流处理，以得到合流音视频流，包括：
    确定所述直播连接确认操作的操作时刻；
    确定所述第一直播音视频流中与所述操作时刻对应的第一直播音视频帧以及所述第二直播音视频流中与所述操作时刻对应的第二直播音视频帧；
    以所述第一直播音视频帧和所述对应的第二直播音视频帧为起始帧，对所述第二直播音视频流和所述第一直播音视频流中的对应音视频帧执行合流处理，以得到合流音视频流。
  12. 如权利要求9所述的方法，还包括：
    响应于针对所述直播连接请求的直播连接取消操作，停止执行向第一服务端推送所述第二客户端的第二直播音视频流。
  13. 一种直播音视频流的处理装置，其中，所述装置应用在第一客户端，包括：
    发送模块，用于响应于与第二客户端的直播连接操作，发送对所述第二客户端的直播连接请求；
    第一接收模块，用于接收与所述第二客户端对应的第二直播音视频流；
    第一合流播放模块，用于响应于接收到的所述第二客户端对应的直播连接通过消息，播放合流音视频流，其中，所述合流音视频流是根据所述第二客户端对应的第二直播音视频流和所述第一客户端对应的第一直播音视频流进行合流处理得到的，其中，所述直播连接通过消息为针对所述直播连接请求的通过消息。
  14. 一种直播音视频流的处理装置，其中，所述装置应用在第二客户端，包括：
    第二接收模块，用于响应于第一客户端的直播连接请求，接收所述第一客户端对应的第一直播音视频流；
    反馈模块，用于响应于直播连接确认操作，反馈直播连接通过消息；
    第二合流播放模块，用于响应于直播连接确认操作，播放合流音视频流，其中，所述合流音视频流是根据所述第一客户端对应的第一直播音视频流和所述第二客户端对应的第二直播音视频流进行合流处理得到的，其中，所述直播连接确认操作为针对所述直播连接请求的确认连接操作，所述直播连接通过消息为针对所述直播连接请求的通过消息。
  15. 一种电子设备，其中，所述电子设备包括：
    处理器；
    用于存储所述处理器可执行指令的存储器；
    所述处理器，用于从所述存储器中读取所述可执行指令，并执行所述可执行指令以实现上述权利要求1-12中任一所述的直播音视频流的处理方法。
  16. 一种计算机可读存储介质，其中，所述计算机可读存储介质存储有计算机程序，所述计算机程序用于执行上述权利要求1-12中任一所述的直播音视频流的处理方法。
  17. 一种计算机程序产品，其中，当所述计算机程序产品中的指令由处理器执行时，实现如权利要求1-12中任一所述的直播音视频流的处理方法。
  18. 一种计算机程序，包括：
    指令，所述指令当由处理器执行时使所述处理器执行如权利要求1-12中任一所述的直播音视频流的处理方法。
PCT/CN2023/085866 2022-04-14 2023-04-03 直播音视频流的处理方法、装置、设备及介质 Ceased WO2023197897A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/856,859 US20250260846A1 (en) 2022-04-14 2023-04-03 Method and apparatus for processing live audio and video stream, and device and medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210395011.9A CN116962803A (zh) 2022-04-14 2022-04-14 直播音视频流的处理方法、装置、设备及介质
CN202210395011.9 2022-04-14

Publications (1)

Publication Number Publication Date
WO2023197897A1 true WO2023197897A1 (zh) 2023-10-19

Family

ID=88328886

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/085866 Ceased WO2023197897A1 (zh) 2022-04-14 2023-04-03 直播音视频流的处理方法、装置、设备及介质

Country Status (3)

Country Link
US (1) US20250260846A1 (zh)
CN (1) CN116962803A (zh)
WO (1) WO2023197897A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100851634B1 (ko) * 2008-02-05 2008-08-13 주식회사 셀런 라이브 멀티미디어 스트림을 풀방식으로 스트리밍하는 방법및 시스템
CN109688419A (zh) * 2018-12-27 2019-04-26 北京潘达互娱科技有限公司 一种直播中的连麦方法、装置及服务器
CN110149527A (zh) * 2019-06-10 2019-08-20 武汉鲨鱼网络直播技术有限公司 线下赛的直播方法、后台服务器及合流服务器
CN111050185A (zh) * 2018-10-15 2020-04-21 武汉斗鱼网络科技有限公司 直播间连麦视频混流方法、存储介质、电子设备及系统
CN111836074A (zh) * 2020-07-23 2020-10-27 北京达佳互联信息技术有限公司 一种连麦直播方法、装置及系统
CN112312154A (zh) * 2020-10-30 2021-02-02 广州华多网络科技有限公司 网络直播控制、执行方法及其相应的装置、设备、介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170163761A1 (en) * 2015-12-07 2017-06-08 Le Holdings (Beijing) Co., Ltd. Method, device and system for obtaining live video
US10762675B2 (en) * 2016-12-12 2020-09-01 Facebook, Inc. Systems and methods for interactive broadcasting
US10192553B1 (en) * 2016-12-20 2019-01-29 Amazon Technologes, Inc. Initiating device speech activity monitoring for communication sessions
US10237581B2 (en) * 2016-12-30 2019-03-19 Facebook, Inc. Presentation of composite streams to users
US11056014B1 (en) * 2020-07-07 2021-07-06 ClassEDU Inc. Virtual classroom over a group meeting platform
US11683538B2 (en) * 2020-09-03 2023-06-20 Meta Platforms, Inc. Live group video streaming

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100851634B1 (ko) * 2008-02-05 2008-08-13 주식회사 셀런 라이브 멀티미디어 스트림을 풀방식으로 스트리밍하는 방법및 시스템
CN111050185A (zh) * 2018-10-15 2020-04-21 武汉斗鱼网络科技有限公司 直播间连麦视频混流方法、存储介质、电子设备及系统
CN109688419A (zh) * 2018-12-27 2019-04-26 北京潘达互娱科技有限公司 一种直播中的连麦方法、装置及服务器
CN110149527A (zh) * 2019-06-10 2019-08-20 武汉鲨鱼网络直播技术有限公司 线下赛的直播方法、后台服务器及合流服务器
CN111836074A (zh) * 2020-07-23 2020-10-27 北京达佳互联信息技术有限公司 一种连麦直播方法、装置及系统
CN112312154A (zh) * 2020-10-30 2021-02-02 广州华多网络科技有限公司 网络直播控制、执行方法及其相应的装置、设备、介质

Also Published As

Publication number Publication date
CN116962803A (zh) 2023-10-27
US20250260846A1 (en) 2025-08-14

Similar Documents

Publication Publication Date Title
CN111064987B (zh) 信息展示方法、装置及电子设备
US20190158630A1 (en) Publication of Applications Using Server-Side Virtual Screen Change Capture
CN111629251B (zh) 视频播放方法、装置、存储介质和电子设备
US12592014B2 (en) Media content processing method and apparatus, device, and storage medium
US11758087B2 (en) Multimedia conference data processing method and apparatus, and electronic device
US20150117513A1 (en) Bandwidth reduction system and method
WO2021249323A1 (zh) 一种信息处理方法、系统、装置、电子设备及存储介质
WO2021254168A1 (zh) 会议控制方法、装置、可读介质及电子设备
CN112306595A (zh) 交互方法、装置和电子设备
WO2019062667A1 (zh) 会议内容的传输方法及装置
WO2023045973A1 (zh) 直播礼物云渲染方法、装置、电子设备及存储介质
WO2021180198A1 (zh) 信息交互方法、装置、电子设备及存储介质
CN114443201A (zh) 消息展示方法、装置、设备及存储介质
WO2023103861A1 (zh) 投屏方法、投屏装置、投屏显示装置、投屏系统及介质
JP2025516284A (ja) マルチメディアデータ処理方法、装置、機器、記憶媒体及びプログラム製品
CN111818383B (zh) 视频数据的生成方法、系统、装置、电子设备及存储介质
CN112291502A (zh) 信息交互方法、装置、系统和电子设备
CN115396684B (zh) 一种连麦展示方法、装置、电子设备、计算机可读介质
CN114489891A (zh) 云应用程序的控制方法、系统、装置、可读介质和设备
WO2023197897A1 (zh) 直播音视频流的处理方法、装置、设备及介质
CN114697149B (zh) 一种视频会议的视图刷新方法、装置、设备及介质
CN112866622B (zh) 信息处理方法、装置、服务器、存储介质和系统
CN115767002A (zh) 一种信息显示方法、装置、设备及介质
CN112256222A (zh) 信息显示方法、装置和电子设备
WO2025139330A1 (zh) 多媒体数据流处理方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23787542

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18856859

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 31/01/2025)

WWP Wipo information: published in national office

Ref document number: 18856859

Country of ref document: US

122 Ep: pct application non-entry in european phase

Ref document number: 23787542

Country of ref document: EP

Kind code of ref document: A1