WO2023011408A1

WO2023011408A1 - Multi-window video communication method, device and system

Info

Publication number: WO2023011408A1
Application number: PCT/CN2022/109423
Authority: WO
Inventors: 张帮明
Original assignee: 华为技术有限公司
Priority date: 2021-08-03
Filing date: 2022-08-01
Publication date: 2023-02-09
Also published as: CN115706829A

Abstract

The present application relates to the technical field of communications, and discloses a multi-window video communication method, device and system. In a multiparty video call process, bandwidth resources can be fully used to avoid network congestion, thereby ensuring the fluency and/or definition of a video. In the present application, multiple windows are displayed on an interface of a receiving end, and when bandwidth resources are insufficient to meet the bandwidth requirements of multiple audio/video streams (i.e., in a weak network environment), the receiving end can perform downgrade subscription according to specific priorities corresponding to the multiple windows, such as reducing the definition of a video, unsubscribing from a video stream or delaying the subscription to a video stream, such that network congestion is avoided, and the fluency and/or definition of a high-priority video is ensured.

Description

A multi-window video communication method, device and system

This application claims the priority of the Chinese patent application with the application number 202110887044.0 and the application title "A Multi-window Video Communication Method, Device and System" submitted to the State Intellectual Property Office on August 3, 2021, the entire contents of which are incorporated by reference incorporated in this application.

technical field

The embodiments of the present application relate to the field of communication technologies, and in particular, to a multi-window video communication method, device, and system.

Background technique

With the popularity of online learning, online meetings, and online chats, the application scenarios of multi-window video communication are becoming more and more diverse. For example, multi-window video communication is used in professional conference scenarios (such as multi-party conference scenarios), life scenarios (such as group video chat scenarios) or online education scenarios in the form of multi-party video calls.

In multi-window video communication, as an implementation solution, the sending end can forward the audio and video streams to the receiving end through the cloud. The cloud side forwards the audio and video streams from the sending end to the receiving end, and performs bandwidth prediction on the communication link for forwarding the audio and video streams. Among them, the bandwidth predicted by the cloud side is used to make communication decisions (such as frame extraction decisions, etc.) when the network is poor.

It can be understood that, when the network is poor, the above-mentioned conventional technology treats multiple communication parties equally. However, different multi-window video communication scenarios have different emphases. For example, in online education scenarios, it is necessary to give priority to ensuring the streaming of courseware/whiteboards, followed by the lecturer’s portrait screen; and in multi-party conference scenarios, it is necessary to prioritize the guarantee of the current speaker (such as voice volume). The video and audio of the person with the largest value), followed by other conference participants. Therefore, based on the above-mentioned conventional technology, when the network is poor, the receiving end cannot adapt to the specific needs, which will cause important video to be stuck when the network is weak. pause.

Contents of the invention

The present application provides a multi-window video communication method, device and system, which can make full use of bandwidth resources to avoid network congestion during a multi-party video call, thereby ensuring smoothness and/or clarity of the video.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

In the first aspect, a multi-window video communication method is provided, the method is applied in the process of video calling between a plurality of first devices and a second device, the method includes: the second device receives a plurality of video messages from a plurality of first devices respectively Audio and video streams; wherein, the audio and video corresponding to the above multiple audio and video streams are respectively played in multiple windows on the interface of the second device; the second device determines that the bandwidth resources are not enough to meet the bandwidth requirements of multiple audio and video streams; the second The second device adjusts the subscription strategy for audio and video streams in one or more windows according to the priorities corresponding to the multiple windows. Exemplarily, the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.

In the solution provided by the first aspect above, when the bandwidth resources at the receiving end are not enough to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams When the bandwidth is required, you can downgrade the subscription according to the specific priorities corresponding to multiple windows, such as reducing video clarity, unsubscribing video streams, or delaying subscribing video streams, etc., to avoid network congestion and ensure the smoothness of high-priority videos and/or clarity. Among them, the priority corresponding to the window can be used to represent specific business requirements or user preferences.

In a possible implementation manner, the above-mentioned second device receiving multiple audio and video streams respectively from multiple first devices includes: the second device receiving multiple audio and video streams respectively from multiple first devices forwarded by a third device Audio and video streaming. As an example, the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.

In a possible implementation manner, the above method further includes: the above-mentioned second device receives multiple bandwidth prediction results for multiple links from the third device; the second device determines that the bandwidth resource is insufficient to meet the Bandwidth requirements for multiple audio and video streams. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided in the present application, the third device may perform bandwidth prediction. For example, the third device may perform bandwidth prediction when forwarding audio and video streams to the second device. Through this solution, the compatibility of the method provided by this application with different network architectures can be improved.

In a possible implementation manner, the above method further includes: the second device measures and obtains multiple bandwidth prediction results for multiple links; the second device determines that the bandwidth resource is insufficient to meet the above multiple Bandwidth requirements for audio and video streams. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided by the present application, the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.

In a possible implementation manner, the first window plays the corresponding audio and video stream at the first definition; the first window is among the above-mentioned multiple windows; the second device adjusts one or more The subscription strategy for audio and video streams in a window includes: the second device subscribes to an audio and video stream with a second definition for the first window; and the second definition is smaller than the first definition. In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, after the second device subscribes to the audio and video stream of the second definition for the first window, the above method further includes: when the first preset condition is met, the second device Subscribe for audio and video streams in first definition. For example, the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.

In a possible implementation manner, after the second device subscribes to the audio and video stream of the second definition for the first window, the above method further includes: when the first preset condition is met for a preset time period, the second device Subscribe to the first definition audio and video stream for the first window. For example, the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.

In a possible implementation manner, the second window plays audio and video streams with a second definition, the second definition is less than or equal to a preset value, and the second window is a window with the lowest priority among the plurality of windows; The second device adjusts the subscription strategy for the audio and video streams in one or more windows according to the priorities corresponding to the multiple windows, including: the second device unsubscribes the video stream corresponding to the second window. In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, after the second window unsubscribes, the above method further includes: the second device displays a mask on the second window. By displaying the mask layer, users can be reminded that they are currently in a weak network environment and user experience can be improved.

In a possible implementation manner, after the second device unsubscribes from the video stream corresponding to the second window, the above method further includes: when a second preset condition is met, the second device resumes subscribing to the second video stream for the second window. high-definition video streaming. For example, the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation manner, after the second device unsubscribes from the video stream corresponding to the second window, the above method further includes: when the second preset condition is met for a preset time period, the second device is the second window Resume subscription to second definition video stream. For example, the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation manner, the priorities corresponding to the multiple windows are determined by the second device according to one or more of the following: the initial volume of the audio corresponding to the multiple windows; the playback volume of the audio corresponding to the multiple windows ; The function of business in multiple windows. Wherein, the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream. In this application, various window priority settings can be supported. For example, the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the priorities corresponding to the foregoing multiple windows are determined by the second device according to user-defined specified operations. In this application, various window priority settings can be supported. For example, it can be user-defined. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the above-mentioned second device receiving multiple audio and video streams from multiple first devices forwarded by the third device includes: the second device receives the first audio and video stream from the first cloud device ; The second device receives the second audio-video stream and the third audio-video stream from the second cloud device. As an example, the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks. Architecture compatibility.

In a possible implementation manner, the foregoing third device is a selective forwarding unit (selective forwarding unit, SFU). Through this solution, the applicability of the method provided in this application and the compatibility with different network architectures can be improved.

In a second aspect, an electronic device (such as a second device) is provided, and the electronic device includes: a transceiver unit, configured to receive a plurality of audio and video streams from a plurality of first devices; wherein, the plurality of audio and video streams correspond to The audio and video are respectively played in multiple windows on the interface of the second device; the display unit is used to play the multiple audio and video streams through the multiple windows; the processing unit is used to determine that bandwidth resources are not enough to satisfy multiple audio The bandwidth requirement of the video stream; and adjust the subscription strategy for the audio and video stream in one or more windows according to the priorities corresponding to multiple windows. Exemplarily, the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.

In the solution provided by the above-mentioned second aspect, when the bandwidth resources of the receiving end are insufficient to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams When the bandwidth is required, you can downgrade the subscription according to the specific priorities corresponding to multiple windows, such as reducing video clarity, unsubscribing video streams, or delaying subscribing video streams, etc., to avoid network congestion and ensure the smoothness of high-priority videos and/or clarity. Wherein, the priority corresponding to the window may be used to represent a specific service requirement or a user's preference.

In a possible implementation manner, the above-mentioned transceiving unit is specifically configured to: receive multiple audio and video streams respectively from multiple first devices forwarded by the third device. As an example, the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.

In a possible implementation manner, the transceiver unit is further configured to: receive multiple bandwidth prediction results for multiple links from the third device; the processing unit is further configured to: determine that the bandwidth resource is insufficient according to the multiple bandwidth prediction results To meet the bandwidth requirements of multiple audio and video streams. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided in the present application, the third device may perform bandwidth prediction. For example, the third device may perform bandwidth prediction when forwarding audio and video streams to the second device. Through this solution, the compatibility of the method provided by this application with different network architectures can be improved.

In a possible implementation manner, the above processing unit is further configured to: measure and obtain multiple bandwidth prediction results for multiple links; and determine that bandwidth resources are insufficient to meet the above multiple audio and video frequency requirements according to the above multiple bandwidth prediction results. The bandwidth requirements of the stream. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided by the present application, the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.

In a possible implementation manner, the first window plays the corresponding audio and video stream at the first definition; the first window is among the above-mentioned multiple windows; the above-mentioned processing unit is specifically configured to: subscribe the first window to the second definition audio and video stream; the second definition is smaller than the first definition. In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, the above-mentioned processing unit is further configured to, when the first preset condition is satisfied, subscribe the first window to the audio-video stream of the first definition. For example, the processing unit may determine whether the first preset condition is met by monitoring the predicted value of the total bandwidth. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.

In a possible implementation manner, the above-mentioned processing unit is further configured to: when the first preset condition is met for a preset time period, subscribe to the first window for the audio and video stream of the first definition. For example, the processing unit may monitor whether the first preset condition is met by monitoring the total bandwidth prediction value. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.

In a possible implementation manner, the second window plays audio and video streams with a second definition, the second definition is less than or equal to a preset value, and the second window is a window with the lowest priority among the plurality of windows; The above processing unit is specifically configured to: unsubscribe from the video stream corresponding to the second window. In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, after the second window unsubscribes, the display unit is further configured to: display a mask on the second window.

In a possible implementation manner, the above-mentioned processing unit is further configured to: resume subscribing to the video stream of the second definition for the second window when the second preset condition is satisfied. For example, the processing unit may determine whether the second preset condition is satisfied by monitoring the total bandwidth prediction value. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation manner, the processing unit is further configured to: when the second preset condition is met for a preset time period, the processing unit resumes subscribing to the video stream of the second definition for the second window. For example, the processing unit may determine whether the second preset condition is satisfied by monitoring the total bandwidth prediction value. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation, the above processing unit is further configured to determine the priorities corresponding to multiple windows according to one or more of the following: initial volume of audio corresponding to multiple windows; playback of audio corresponding to multiple windows Volume; a function of business in multiple windows. Wherein, the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream. In this application, various window priority settings can be supported. For example, the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the above processing unit is further configured to determine priorities corresponding to multiple windows according to user-defined specified operations. In this application, various window priority settings can be supported. For example, may be user-defined. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the above-mentioned transceiving unit is specifically configured to: receive the first audio-video stream from the first cloud device; and receive the second audio-video stream and the third audio-video stream from the second cloud device. As an example, the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks. Architecture compatibility.

In a possible implementation manner, the foregoing third device is an SFU. Through this solution, the applicability of the method provided in this application and the compatibility with different network architectures can be improved.

In a third aspect, an electronic device (such as a second device) is provided, and the electronic device includes: a memory for storing a computer program; a transceiver for receiving or sending a radio signal; a display for displaying an interface; a processor, It is used to execute the computer program, so that the electronic device receives multiple audio and video streams from multiple first devices through a transceiver; wherein, the audio and video corresponding to the multiple audio and video streams are respectively displayed on the interface of the second device Play in multiple windows; determine that the bandwidth resources are not enough to meet the bandwidth requirements of multiple audio and video streams; and adjust the subscription strategy for the audio and video streams in one or more windows according to the priorities corresponding to the multiple windows. Exemplarily, the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.

In the solution provided by the third aspect above, when the bandwidth resources at the receiving end are insufficient to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams When the bandwidth is required, you can downgrade the subscription according to the specific priorities corresponding to multiple windows, such as reducing video clarity, unsubscribing video streams, or delaying subscribing video streams, etc., to avoid network congestion and ensure the smoothness of high-priority videos and/or clarity. Wherein, the priority corresponding to the window may be used to represent a specific service requirement or a user's preference.

In a possible implementation manner, the foregoing transceiver is specifically configured to: receive multiple audio and video streams respectively from multiple first devices forwarded by the third device. As an example, the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.

In a possible implementation manner, the transceiver is further configured to: receive multiple bandwidth prediction results for multiple links from the third device; the processor is further configured to: determine that bandwidth resources are insufficient according to the multiple bandwidth prediction results To meet the bandwidth requirements of multiple audio and video streams. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided in the present application, the third device may perform bandwidth prediction. For example, the third device may perform bandwidth prediction when forwarding audio and video streams to the second device. Through this solution, the compatibility of the method provided by this application with different network architectures can be improved.

In a possible implementation manner, the above-mentioned processor is further configured to: measure and obtain multiple bandwidth prediction results for multiple links; The bandwidth requirements of the stream. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided by the present application, the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.

In a possible implementation manner, the first window plays the corresponding audio and video stream at the first definition; the first window is among the plurality of windows; the processor is specifically configured to: subscribe the first window to the second definition audio and video stream; the second definition is smaller than the first definition. In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, the above-mentioned processor is further configured to: when the first preset condition is satisfied, subscribe the first window to the audio-video stream of the first definition. For example, the processor may monitor the total bandwidth prediction value to determine whether the first preset condition is met. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.

In a possible implementation manner, the above-mentioned processor is further configured to: subscribe for the first window to an audio and video stream of the first definition when the first preset condition is met for a preset time period. For example, the processor may monitor the total bandwidth prediction value to determine whether the first preset condition is met. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.

In a possible implementation manner, the second window plays audio and video streams with a second definition, the second definition is less than or equal to a preset value, and the second window is a window with the lowest priority among the plurality of windows; The processor above is specifically configured to: unsubscribe from the video stream corresponding to the second window. In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, after the second window unsubscribes, the display is further configured to display a mask on the second window.

In a possible implementation manner, the above-mentioned processor is further configured to: resume subscribing to the video stream of the second definition for the second window when the second preset condition is met. For example, the processor may monitor the total bandwidth prediction value to determine whether the second preset condition is met. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation manner, the processor is further configured to: when the second preset condition is met for a preset time period, the processing unit resumes subscribing to the video stream of the second definition for the second window. For example, the processor may monitor the total bandwidth prediction value to determine whether the second preset condition is met. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation, the above-mentioned processor is further configured to determine the priorities corresponding to multiple windows according to one or more of the following: the initial volume of the audio corresponding to the multiple windows; the playback of the audio corresponding to the multiple windows Volume; a function of business in multiple windows. Wherein, the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream. In this application, various window priority settings can be supported. For example, the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the above-mentioned processor is further configured to: determine priorities corresponding to multiple windows according to user-defined specified operations. In this application, various window priority settings can be supported. For example, it can be user-defined. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the above-mentioned transceiver is specifically configured to: receive the first audio-video stream from the first cloud device; and receive the second audio-video stream and the third audio-video stream from the second cloud device. As an example, the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks. Architecture compatibility.

In a fourth aspect, there is provided a multi-window video communication method, which is applied in the process of a video call between multiple first devices and a second device in a communication system, and the method includes: multiple first devices send to the second device Audio and video streams; wherein, the audio and video corresponding to the above multiple audio and video streams are respectively played in multiple windows on the interface of the second device; the second device determines that the bandwidth resources are not enough to meet the bandwidth requirements of multiple audio and video streams; the second The second device adjusts the subscription strategy for audio and video streams in one or more windows according to the priorities corresponding to the multiple windows. Exemplarily, the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.

In the solution provided by the fourth aspect above, when the bandwidth resources at the receiving end are insufficient to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams When the bandwidth is required, you can downgrade the subscription according to the specific priorities corresponding to multiple windows, such as reducing video clarity, unsubscribing video streams, or delaying subscribing video streams, etc., to avoid network congestion and ensure the smoothness of high-priority videos and/or clarity. Wherein, the priority corresponding to the window may be used to represent a specific service requirement or a user's preference.

In a possible implementation manner, the above-mentioned communication system further includes: one or more third devices, configured to receive the above-mentioned multiple audio and video streams from the above-mentioned multiple first devices, and forward the above-mentioned multiple audio and video streams to the second device video stream. As an example, the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.

In a possible implementation manner, the third device is further configured to: measure and obtain multiple bandwidth prediction results for multiple links during the process of forwarding the multiple audio and video streams to the second device; The device is specifically configured to: determine, according to multiple bandwidth prediction results, that bandwidth resources are insufficient to meet bandwidth requirements of multiple audio and video streams. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided in the present application, the third device may perform bandwidth prediction. For example, the third device may perform bandwidth prediction when forwarding audio and video streams to the second device. Through this solution, the compatibility of the method provided by this application with different network architectures can be improved.

In a possible implementation manner, the second device is further configured to: during the process of receiving the above multiple audio and video streams, perform measurement to obtain multiple bandwidth prediction results for multiple links; the second device is specifically configured to: It is determined according to the multiple bandwidth prediction results that the bandwidth resources are insufficient to meet the bandwidth requirements of the multiple audio and video streams. Wherein, the above multiple links correspond to the above multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams. As an example, in the multi-window video communication method provided by the present application, the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.

In a possible implementation manner, the above-mentioned second device is further configured to: play the corresponding audio and video stream with the first definition through the first window; the first window is among the above-mentioned multiple windows; the second device The corresponding priority adjustment of the subscription strategy for the audio and video streams in one or more windows includes: the second device subscribes to the audio and video streams of the second definition for the first window; the second definition is lower than the first definition. In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, after the second device subscribes to the audio and video stream of the second definition for the first window, the second device is further configured to: when the first preset condition is met, Subscribe for audio and video streams in first definition. For example, the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.

In a possible implementation manner, after the second device subscribes to the audio and video stream of the second definition for the first window, the second device is further configured to: when the first preset condition is met for a preset time period, Subscribe to the first definition audio and video stream for the first window. For example, the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth. For example, the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when meeting the bandwidth requirements for restoring clarity, restore the definition of the degraded video to maximize the fluency and/or clarity of the video

In a possible implementation manner, the second device is further configured to: play audio and video streams with a second definition through a second window, the second definition is less than or equal to a preset value, and the second window is the above-mentioned multiple Among the windows, the window with the lowest priority; the second device adjusts the subscription strategy for the audio and video streams in one or more windows according to the priorities corresponding to the multiple windows, including: the second device unsubscribes the video stream corresponding to the second window . In this application, when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.

In a possible implementation manner, after the second window unsubscribes, the second device is further configured to: display a mask on the second window. By displaying the mask layer, users can be reminded that they are currently in a weak network environment and user experience can be improved.

In a possible implementation manner, after the second device unsubscribes from the video stream corresponding to the second window, the second device is further configured to: resume subscribing to the second window for the second window when the second preset condition is met. high-definition video streaming. For example, the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation manner, after the second device unsubscribes from the video stream corresponding to the second window, the second device is further configured to: when the second preset condition is met for a preset time period, set Resume subscription to second definition video stream. For example, the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth. For example, the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition. In this application, the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.

In a possible implementation manner, the second device is further configured to determine the priorities corresponding to the multiple windows according to one or more of the following: the initial volume of the audio corresponding to the multiple windows; The playback volume of the audio corresponding to the window; the function of business in multiple windows. Wherein, the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream. In this application, various window priority settings can be supported. For example, the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the above-mentioned second device is further configured to: determine priorities corresponding to multiple windows according to user-defined specified operations. In this application, various window priority settings can be supported. For example, it can be user-defined. The diversified window priority setting can facilitate diversified operations of the user and improve user experience.

In a possible implementation manner, the above-mentioned one or more third devices include a first cloud device and a second cloud device, wherein the first cloud device is used to forward the first audio and video stream to the second device, and the second cloud device The device is configured to forward the second audio-video stream and the third audio-video stream to the second device. As an example, the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks. Architecture compatibility.

In a fifth aspect, a communication system is provided, and the communication system includes: a plurality of first devices and the electronic device in any possible implementation manner of the second aspect or the third aspect. Wherein, the above-mentioned multiple first devices send audio and video streams to the second device for playing in multiple windows on the interface of the second device respectively. The communication system is used to implement the method in any possible implementation manner of the fourth aspect.

In a possible implementation manner, the foregoing communication system further includes: one or more third devices, configured to forward audio and video streams from the plurality of first devices to an electronic device.

According to a sixth aspect, a computer-readable storage medium is provided. Computer program code is stored on the computer-readable storage medium. When the computer program code is executed by a processor, the processor can realize any possible implementation of the first aspect. methods in methods.

In a seventh aspect, there is provided a chip system, the chip system includes a processor and a memory, and computer program code is stored in the memory; when the computer program code is executed by the processor, the processor implements any one of the first aspect. method in one possible implementation. The system-on-a-chip may consist of chips, or may include chips and other discrete devices.

In an eighth aspect, a computer program product is provided, the computer program product comprising computer instructions. When the computer instructions are run on the computer, the computer is made to implement the method in any possible implementation manner of the first aspect.

Description of drawings

Fig. 1 is an exemplary diagram of a selective forwarding unit (selective forwarding unit, SFU) forwarding scheme provided by the embodiment of the present application;

FIG. 2 is a video communication interaction diagram based on SFU frame extraction provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a layered encoding provided by an embodiment of the present application;

FIG. 4 is an example diagram of a multi-window video communication scenario provided by an embodiment of the present application;

FIG. 5A is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application;

FIG. 5B is a software architecture diagram of an electronic device provided by an embodiment of the present application;

6 is a schematic diagram of a call service architecture based on real time communication (real time communication, RTC) provided by the embodiment of the present application;

FIG. 7 is a structure diagram of a multi-window video communication provided by an embodiment of the present application;

FIG. 8 is a flowchart of a multi-window video communication method provided by an embodiment of the present application;

FIG. 9 is an interaction diagram of a multi-window video communication method provided by an embodiment of the present application;

FIG. 10 is an example diagram of three types of multi-window display provided by the embodiment of the present application;

FIG. 11 is an example diagram of a multi-window display provided by the embodiment of the present application;

Fig. 12 is an example diagram of displaying a mask when a weak network is provided by the embodiment of the present application;

FIG. 13 is a flow chart of another multi-window video communication method provided by the embodiment of the present application;

FIG. 14 is an example diagram of displaying a weak network prompt when a weak network is provided by an embodiment of the present application;

FIG. 15 is an example diagram 1 of a method for setting a priority corresponding to a window provided in an embodiment of the present application;

Figure 16 is an example of the setting method of the priority corresponding to the window provided in the embodiment of the present application Figure 2;

FIG. 17 is a structural block diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Among them, in the description of the embodiments of this application, unless otherwise specified, "/" means or means, for example, A/B can mean A or B; "and/or" in this article is only a description of associated objects The association relationship of indicates that there may be three kinds of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. In addition, in the description of the embodiments of the present application, "plurality" refers to two or more than two.

Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

An embodiment of the present application provides a multi-window video communication method, which is applied to a process in which multiple users conduct a multi-party real-time video call.

As an example, a multi-window video communication method provided in this embodiment of the application can be applied to a multi-party conference scenario. For example, user A, user B, and user C join a conference for a multi-party real-time video call, where user A is the conference speaker, and user B and user C are conference participants.

As another example, the multi-window video communication method provided in the embodiment of the present application may be applied to a group video chat scene. For example, user A, user B, and user C join a group chat for a multi-party real-time video call, wherein during the group chat, user A, user B, and user C all speak freely.

As another example, the multi-window video communication method provided in the embodiment of the present application may be applied to an online education scenario. For example, user A, user B, and user C join an online class group for a multi-party real-time video call, where user A is a teacher, and user B and user C are students. User A shows courseware/whiteboard to user B and user C while teaching and explaining.

In multi-window video communication scenarios such as the above-mentioned multi-party conference scenario, group video chat scenario, or online education scenario, as an implementation solution, the sending end (such as the first device) can send a message to the receiving end through a third device (such as a cloud device) (such as the second device) forward audio and video streams. For example, the third device may complete the forwarding of the audio and video stream based on the publishing and subscription relationship of the audio and video stream. Wherein, the third device supports the end-to-end encryption feature, and does not need to analyze the audio and video streams at the sending end. Exemplarily, the sending end and the receiving end store keys (such as a public key and a private key), which are not known by the third device, so the third device cannot parse the user's audio and video streams, which is highly secure.

Exemplarily, the sending end may forward the audio and video streams to the receiving end through a selective forwarding unit (selective forwarding unit, SFU). In some embodiments, multiple SFUs can be deployed in a distributed manner to improve network scalability. Please refer to FIG. 1 , which shows an example diagram of an SFU forwarding solution. As shown in Figure 1, device 1 and device 2 respectively select SFU 1 and SFU 2 to transmit audio and video streams to device 4, and device 3 selects SFU3 to send audio and video streams to device 4 and device 5.

In some embodiments, in the SFU forwarding scheme, the sending end can select the optimal SFU for audio and video stream forwarding. For example, the sending end may select the optimal SFU according to network conditions (such as operators, regions, etc.), which is not limited in this application.

Further, in the above forwarding scheme, the third device (such as a cloud device) can also perform bandwidth prediction on the downlink of the forwarded audio and video stream when forwarding, and send the prediction result to the receiving end for use by the receiving end in the network. Time-lapse communication decisions.

Taking video layered coding as an example, a third device (such as an SFU) can make frame extraction decisions when the network is poor. Among them, frame extraction can reduce downlink bandwidth consumption, avoid network congestion, and ensure audio and video fluency and/or clarity in a weak network environment.

Please refer to FIG. 2 . FIG. 2 takes the bandwidth prediction performed by the SFU through the bandwidth prediction module as an example, and shows a video communication interaction diagram based on SFU frame extraction. As shown in Figure 2, the video communication process based on SFU frame extraction mainly includes the following five steps:

Step 1: The sending end collects, encodes and encrypts audio and video data;

Step 2: The sender sends the frame data, frame type, etc. to the SFU;

Step 3: The forwarding module of the SFU forwards the frame data to the receiving end, and at the same time, the bandwidth prediction module of the SFU performs downlink bandwidth prediction;

Step 4: SFU makes a frame drawing decision based on the bandwidth prediction value;

Step 5: After receiving the frame data, the receiving end performs decryption, decoding and rendering.

In some examples, the bandwidth prediction performed by the bandwidth prediction module may include: the downlink bandwidth prediction value calculated by the bandwidth prediction module according to data such as time delay and packet loss of downlink data transmission. For the specific method of downlink bandwidth prediction, reference may be made to conventional technologies, which are not limited in this application.

Taking layered encoding as an example, as shown in Figure 3, when encoding continuous dynamic image frames in a video stream, the continuous image frames can be compressed into instant decoding refresh (instantaneous decoding refresh, IDR) frames, forward predictive encoding There are three types of frame (P frame) and bidirectional predictive interpolation coded frame (B frame). Wherein, FIG. 3 takes group of pictures (group of pictures, GOP)=30 as an example, wherein, GOP is a group of continuous pictures. In some embodiments, IDR frame compression can be achieved 6:1 without any perceptible blurring. Using P frame compression at the same time as IDR frame compression can achieve a higher compression ratio without noticeable blurring. B frame compression can achieve a compression ratio of 200:1, and its file size is generally 15% of the IDR frame compression size, less than half of the P frame compression size. Among them, the IDR frame compression can remove the spatial redundancy of the image, and the P frame and B frame compression can remove the temporal redundancy.

Wherein, the IDR frame compression adopts full-frame compression encoding, that is, the full-frame image information is subjected to joint photographic experts group (JPEG) compression encoding. The IDR frame describes the details of the image background and moving subject, therefore, the IDR frame is also called a key frame. Based on this, IDR frames can be decoded and rendered independently. When decoding a video stream, only the data of the IDR frame can be used to reconstruct the complete picture.

The IDR frame can be used as a reference for P and B frames. For example, an IDR frame can be used as a reference for a P frame and a B frame after performing intra prediction, residual determination, residual transformation and quantization, variable length coding and arithmetic coding, image reconstruction and filtering, respectively. Wherein, the residual can be determined by subtracting the predicted value from the pixel value.

The P frame is a coded frame separated by 1 to 2 frames behind the IDR frame. The P frame belongs to the interframe coding of forward prediction, so it only refers to the IDR frame or P frame closest to it for prediction. For example, the P frame adopts the method of motion compensation to predict the difference and the motion vector between the current frame and the previous nearest IDR frame or P frame. When decoding a P frame, the complete P frame image must be reconstructed after summing the prediction value and the prediction error in the IDR frame. As shown in Figure 3, a P frame is predicted based on its preceding IDR frame. The P frame can be the reference frame of the P frame behind it, or the reference frame of the B frame before and after it.

B-frames are bidirectionally inter-coded. B frames extract data from preceding and following IDR frames or P frames. The B frame is compressed based on the difference between the current frame and the image of the previous frame and the next frame to complete decoding and rendering. For example, the B frame predicts the prediction error and motion vector between the current frame and the preceding IDR frame or between the P frame and the following P frame. As shown in Figure 3, a B frame is predicted based on its preceding IDR frame and P frame, or based on its preceding and following P frames. B frames are not referenced by other frames.

It can be understood that in layered coding similar to that shown in FIG. 3 , B frames are not referenced by other frames, so the discarding of B frames will not affect the reference relationship of video frames. Based on this, when the network is poor at the receiving end, it can decide whether to extract B frames according to the bandwidth prediction value and the bandwidth requirements of various types of encoded frames. Among them, the elimination of B frames can achieve the purpose of reducing the downlink bandwidth load.

As an example, in this embodiment of the present application, the bandwidth requirements of various types of coded frames (such as IDR frames, P frames, and B frames) can be obtained by counting the original frame data from the sending end.

Exemplarily, taking the multi-window video communication scenario including large windows and small windows shown in FIG. 4 as an example, Table 1 below shows an example of bandwidth requirements. Among them, the high-definition video is played in the large window, and the resolution of the video in the large window is 540P; the normal definition video is played in the small window, and the resolution of the video in the small window is 360P.

Table 1

As shown in Table 1, the required bandwidth of the video stream played in the large window shown in Figure 4 is 700 kilobits per second (kilobits per second, kbps) before drawing frames; the required bandwidth after drawing B frames is 400kbps. The video streams played in the small window 1 and small window 2 shown in Figure 4 require a bandwidth of 500 kbps before frame extraction; and require a bandwidth of 300 kbps after extracting B frames.

Among them, in the embodiment of this application, the bandwidth required for audio and video streams is affected by the codec (Codec), the code rate control of the upstream bandwidth, the number of layers of layered coding, and the resolution/frame rate of the video. This example does not limit the specific calculation strategies and specific algorithms for the required bandwidth of audio and video streams.

Please refer to Table 2 below, which shows an example of an SFU frame extraction strategy.

Table 2

As shown in Table 2, if the predicted bandwidth satisfies the bandwidth requirements of IDR frames, P frames, and B frames, that is, the predicted bandwidth value ≥ the required bandwidth of IDR frames + the required bandwidth of P frames + the required bandwidth of B frames, then the receiving end makes a decision Does not pump frames.

If the predicted bandwidth meets the bandwidth requirements of IDR frames and P frames after extracting B frames, but does not meet the bandwidth requirements of IDR frames, P frames, and B frames, that is, the required bandwidth of IDR frames + the required bandwidth of P frames ≤ the predicted bandwidth value <Bandwidth required by IDR frame+Bandwidth required by P frame+Bandwidth required by B frame, or the predicted bandwidth does not meet the bandwidth requirements of IDR frame and P frame, that is, the predicted bandwidth value<Bandwidth required by IDR frame+Bandwidth required by P frame , the receiving end decides to draw B frames.

It should be noted that, for the case where the predicted bandwidth value is less than the required bandwidth of the IDR frame + the required bandwidth of the P frame, even if the B frame is extracted, the predicted bandwidth still cannot meet the required bandwidth after the frame is drawn. congestion. Network congestion will increase the delay, resulting in audio and video freezes.

That is to say, taking the bandwidth requirements shown in Table 1 as an example, if the SFU frame extraction strategy shown in Table 2 is adopted, as shown in Table 3, when the predicted bandwidth value is ≥ 1700 kbps, the predicted bandwidth can meet the maximum requirements of the three windows. Bandwidth is required. In this case, the receiving end decides not to draw frames. When 1000kbps≤predicted bandwidth<1700kbps, although the predicted bandwidth does not meet the maximum required bandwidth of the three windows, it can meet the minimum required bandwidth of the three windows. In this case, the receiving end decides to extract B frames. When the bandwidth prediction value is less than 1000kbps, the predicted bandwidth cannot meet the minimum required bandwidth of the three windows. In this case, even if B frames are extracted, the predicted bandwidth still cannot meet the required bandwidth after frame extraction. Therefore, after frame extraction There are still audio and video freezes in the large window, small window 1, and small window 2 shown in Figure 4.

table 3

Based on the above example, it can be understood that when the network is poor, conventional techniques are used to treat multiple communication parties equally. However, due to the different emphases of different multi-window video communication scenarios, for example, multi-party conference scenarios need to give priority to the video and audio of the current speaker (such as the person with the highest voice volume value, and the user corresponding to the large window as shown in Figure 4). , followed by other conference participants; another example is that in online education scenarios, the fluency and/or clarity of courseware/whiteboards must be guaranteed first, followed by the lecturer’s portrait; therefore, when the network is poor, the receiving end cannot Adaptive processing according to specific needs will cause important videos to freeze in a weak network environment.

Taking Table 3 as an example, when 1000kbps≤predicted bandwidth value<1700kbps, the receiver decides to draw B frames for the large window, small window 1, and small window 2 shown in Figure 4, so the large window, small window 1, and small window 2 The smoothness of the video in the medium and low-level videos has been reduced. As another example, when the predicted bandwidth value is less than 1000kbps, the receiver decides to draw B frames for the large window, small window 1, and small window 2 shown in Figure 4, but the predicted bandwidth after frame extraction still cannot meet the minimum requirements of the three windows. Bandwidth is required, so there is network congestion in all three windows. Network congestion will increase the delay, resulting in audio and video freezes in the large window, small window 1, and small window 2 shown in Figure 4.

In order to solve the above problems, an embodiment of the present application provides a multi-window video communication method. In this method, when the receiving end is in a weak network environment, the terminal data stream can be adjusted according to the specific business scenarios and demands of multiple video call users to ensure priority Normal playback of high priority video streams. For example, in the scenario shown in FIG. 4 , priority is given to ensuring video fluency and/or clarity in the large window. As another example, in online education scenarios, priority is given to ensuring the video fluency and/or clarity of courseware/whiteboards.

Wherein, in the embodiment of the present application, a weak network means that bandwidth resources are insufficient to meet the bandwidth requirements of audio and video streams.

Exemplarily, in a scene where the downlink bandwidth is limited or the bandwidth fluctuates greatly, such as a high-speed rail scene, a scene far away from a wireless local area network (wireless local area networks, WLAN) (such as a WiFi network) hotspot, a remote area or a densely populated area, the bandwidth is limited Limiting scenarios, etc., using the method provided by the embodiment of this application to downgrade and subscribe to video streams based on specific priorities can refine the gradient of video stream control, and avoid network congestion while making full use of downlink bandwidth, thereby ensuring maximum protection The smoothness and/or clarity of the video.

The multi-window video communication method provided by the embodiment of the present application can be applied to, but not limited to, smart phones, netbooks, tablet computers, smart watches, smart bracelets, phone watches, smart cameras, palmtop computers, personal computers (personal computers, PCs), personal Digital assistant (personal digital assistant, PDA), portable multimedia player (portable multimedia player, PMP), augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) equipment, TV, projection equipment or human-computer interaction Somatosensory game consoles in the scene, etc. Alternatively, the method may also be applied to electronic devices of other types or structures, which is not limited in this application.

Please refer to FIG. 5A . FIG. 5A shows a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application by taking a smart phone as an example. As shown in Figure 5A, the electronic device can include a processor 510, a memory (comprising an external memory interface 520 and an internal memory 521), a universal serial bus (universal serial bus, USB) interface 530, a charging management module 540, and a power management module 541 , battery 542, antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, audio module 570, speaker 570A, receiver 570B, microphone 570C, earphone jack 570D, sensor module 580, button 590, motor 591, indicator 592 , a camera 593, a display screen 594, and a subscriber identification module (subscriber identification module, SIM) card interface 595, etc. The sensor module 580 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

It can be understood that, the structure shown in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the illustrations, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

Processor 510 may include one or more processing units. For example: the processor 510 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a flight controller, Video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

A memory may also be provided in the processor 510 for storing instructions and data. In some embodiments, the memory in processor 510 is a cache memory. The memory may hold instructions or data that the processor 510 has just used or recycled. If the processor 510 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 510 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 510 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

The charging management module 540 is used for receiving charging input from the charger. The power management module 541 is used for connecting the battery 542 , the charging management module 540 and the processor 510 . The power management module 541 receives the input of the battery 542 and/or the charging management module 540, and supplies power for the processor 510, the internal memory 521, the display screen 594, the camera component 593, and the wireless communication module 560, etc.

The wireless communication function of the electronic device can be realized by the antenna 1, the antenna 2, the mobile communication module 550, the wireless communication module 560, the modem processor and the baseband processor.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 550 can provide wireless communication solutions including 2G/3G/4G/5G applied to electronic devices. The mobile communication module 550 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 550 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 550 can also amplify the signal modulated by the modem processor, convert it into electromagnetic wave and radiate it through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 550 may be set in the processor 510 . In some embodiments, at least part of the functional modules of the mobile communication module 550 and at least part of the modules of the processor 510 may be set in the same device.

In the embodiment of the present application, the electronic device can communicate with the cloud device through the mobile communication module 550, for example, sending audio and video streams and the like.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 570A, a receiver 570B, etc.), or displays an image or video through a display screen 594 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent of the processor 510, and be set in the same device as the mobile communication module 550 or other functional modules.

In this embodiment of the present application, the electronic device may play audio corresponding to multiple windows through an audio device, and play corresponding video through multiple windows on the display screen 594 .

The wireless communication module 560 can provide wireless local area networks (wireless local area networks, WLAN) (such as WiFi network), Bluetooth BT, global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 560 may be one or more devices integrating at least one communication processing module. The wireless communication module 560 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 510 . The wireless communication module 560 can also receive the signal to be transmitted from the processor 510 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 550, and the antenna 2 is coupled to the wireless communication module 560, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device realizes the display function through the GPU, the display screen 594, and the application processor. The GPU is a microprocessor for image processing, connected to the display screen 594 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 510 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 594 is used to display images, videos and the like. Display 594 includes a display panel. In some embodiments, the electronic device may include 1 or N display screens 594, where N is a positive integer greater than 1.

In the embodiment of the present application, the electronic device may render videos in multiple windows through the GPU, and the display screen 594 is used to play corresponding videos through the multiple windows.

The electronic device can realize the shooting function through ISP, camera component 593 , video codec, GPU, display screen 594 and application processor.

The external memory interface 520 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 510 through the external memory interface 520 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 521 may be used to store computer-executable program codes including instructions. The internal memory 521 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data (such as audio data, phone book, etc.) created during the use of the electronic device. In addition, the internal memory 521 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 510 executes various functional applications and data processing of the electronic device by executing instructions stored in the internal memory 521 and/or instructions stored in a memory provided in the processor.

The electronic device can realize the audio function through the audio module 570, the speaker 570A, the receiver 570B, the microphone 570C, and the application processor. Such as music playback, recording, etc. For the specific working principles and functions of the audio module 570, the loudspeaker 570A, the receiver 570B and the microphone 570C, as well as the specific working principles and functions of the buttons 590, the motor 591, the indicator 592 and the SIM card interface 595, you can refer to the introduction in the conventional technology .

It should be noted that the hardware modules included in the electronic device shown in FIG. 5A are only described as examples, and do not limit the specific structure of the electronic device. For example, an electronic device may also include other functional modules.

Taking the end-side device (such as the sender and the receiver) of the Android system including a layered architecture as an example, as shown in FIG. 5B , the software of the electronic device can be divided into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. As shown in Figure 5B, the software structure of the electronic device can be divided into three layers from top to bottom:

Application layer (referred to as application layer), application framework layer (referred to as framework layer), system library, Android runtime and kernel layer (also referred to as driver layer).

Wherein, the application layer may include a series of application packages, such as camera, gallery, calendar, call, map, navigation, bluetooth, music, video, short message and other applications. For convenience of description, the application program is referred to as application for short below. As shown in FIG. 5B, the application layer may also include an RTC software development kit (software development kit, SDK) and a call SDK.

Among them, the call application is mainly used to complete a user interface (user interface, UI) and interaction logic. The call SDK is mainly used to connect with the communication cloud, provide account management, contact management, signaling communication and other capabilities, complete audio and video data collection, broadcasting, encoding and decoding, connect with RTC SDK, and complete the intercommunication of media streams during the call. The RTC SDK is mainly used to interact with the RTC cloud to provide the ability to send media streams (such as audio and video streams).

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. As shown in FIG. 5B, the application framework layer may include a window management server (window manager service, WMS), an activity management server (activity manager service, AMS) and an input event management server (input manager service, IMS). In some embodiments, the application framework layer may also include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. (not shown in FIG. 5B ).

The system library and the Android runtime include the functions that the FWK needs to call, the Android core library, and the Android virtual machine. A system library can include multiple function modules. For example: browser kernel, three-dimensional (3 dimensional, 3D) graphics, font library, etc.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The kernel layer is the layer between hardware and software. The kernel layer can include display drivers, input/output device drivers (for example, keyboards, touch screens, earphones, speakers, microphones, etc.), device nodes, camera drivers, audio drivers, and sensor drivers. The user performs an input operation through the input device, and the kernel layer can generate corresponding original input events according to the input operation, and store them in the device node. Input/output device drivers can detect user input events. For example, an operation where the user sets the window priority by dragging the window.

Further, as shown in FIG. 5B , the end-side device (such as the receiving end) also includes a weak network decision module and a coder/decoder. The weak network decision module is used to predict the overall bandwidth according to the predicted bandwidth, determine whether it is in a weak network environment according to the overall bandwidth and the bandwidth requirements of multiple audio and video streams, and perform weak network decision-making and processing when it is determined to be in a weak network environment.

The encoding/decoding module is responsible for decoding audio and video streams.

It should be noted that, in FIG. 5B , the weak network decision-making module and the encoding/decoding module are located at the application framework layer as an example. In fact, in the embodiment of the present application, the weak network decision module and the codec/decode module can be located at any software architecture layer of the device on the end (such as the receiving end). For example, the above-mentioned weak network decision-making module and encoding/decoding module may also be located at the software architecture layer such as the application program layer, the kernel layer, or the system library of the end-side device (such as the receiving end).

As an example, a multi-window video communication method provided in the embodiment of the present application may be implemented based on a real time communication (real time communication, RTC) call service architecture.

Please refer to FIG. 6, which shows a schematic diagram of an RTC-based call service architecture. As shown in Figure 6, the end-side devices communicate through cloud devices. It should be noted that, in FIG. 6 , the architecture includes two end-side devices as an example, but this embodiment of the present application does not limit the specific number of end-side devices in multi-window video communication.

As shown in Figure 6, cloud devices include communication cloud and RTC cloud. The communication cloud includes an account server and a signaling server. RTC cloud includes RTC server and RTC SFU.

Wherein, the account server is mainly used for storing and maintaining account information, contact information, and push (Push) information. The signaling server is mainly responsible for forwarding call information and call control signaling. The RTC server is mainly responsible for room (Room) access authentication, SFU resource allocation, routing policy, and Room operation/interaction. The RTC SFU is mainly used to maintain the publish/subscribe relationship of media streams, forward media streams (such as audio and video streams), and network adaptability.

Wherein, the end-side device is a sending end (such as a first device) or a receiving end (such as a second device). As shown in Figure 6, the call application, call SDK and RTC SDK are installed in the end-side device.

Among them, the call application is mainly used to complete UI and interaction logic. The call SDK is mainly used to connect with the communication cloud, provide account management, contact management, signaling communication and other capabilities, complete audio and video data collection, broadcasting, encoding and decoding, connect with RTC SDK, and complete the intercommunication of media streams during the call. The RTC SDK is mainly used to interact with the RTC cloud to provide the ability to send media streams (such as audio and video streams).

As shown in FIG. 6 , the end-side device also includes an encoding/decoding module. Exemplarily, when the end-side device is a sending end, the encoding module is used to encode audio and video streams; when the end-side device is a receiving end, the decoding module is used to decode audio and video streams.

As an example, the structure of the RTC SFU in the call service architecture shown in FIG. 6 may be the structure of the SFU shown in FIG. 2 . Wherein, the RTC SFU may include a forwarding module and a bandwidth prediction module. In the embodiment of the present application, the forwarding module is used to forward audio and video streams. The bandwidth prediction module is used for bandwidth prediction.

As shown in FIG. 6 , in order to implement the multi-window video communication method provided by the embodiment of the present application, the terminal-side device (such as the receiving terminal) further includes a weak network decision module. The weak network decision module is used to predict the overall bandwidth (that is, predict bandwidth resources), determine whether it is in a weak network environment according to the overall bandwidth and the bandwidth requirements of multiple audio and video streams, and perform weak network decision-making and processing when it is determined to be in a weak network environment. For the specific role of the weak network decision-making module, you can refer to the specific introduction below.

Taking the call service architecture shown in FIG. 6 as an example, please refer to FIG. 7 , which shows a multi-window video communication architecture diagram provided by an embodiment of the present application.

As shown in Figure 7, RTC SFU 1 and RTC SFU 2 are deployed in a distributed manner to improve network scalability. Each sender shown in Figure 7 can select the optimal RTC SFU for audio and video stream forwarding by itself, for example, select the optimal RTC SFU according to network conditions (such as operators, regions, etc.). For example, as shown in Figure 7, sender A chooses to send audio and video stream 1 to receiver D through RTC SFU 1; sender B and sender C choose to send audio and video stream 2 and audio and video to receiver D through RTC SFU 2 respectively Stream 3.

Among them, the bandwidth prediction module of RTC SFU 1 shown in Figure 7 performs bandwidth prediction according to the downlink data (audio and video stream 1 shown in Figure 7), and sends the bandwidth prediction result to the weak network decision module of the receiving end D. For example, the bandwidth prediction module calculates the bandwidth prediction value 1 according to the data such as time delay and packet loss of the audio and video stream 1 shown in FIG. 7 . Similarly, the bandwidth prediction module of the RTC SFU 2 shown in Figure 7 calculates the bandwidth prediction value 2 according to the data such as the delay and packet loss of the audio and video stream 2 shown in Figure 7, and sends it according to the audio and video stream 3 shown in Figure 7. The bandwidth prediction value 3 is obtained by calculating the time delay, packet loss and other data.

The receiving end D shown in FIG. 7 includes a decoding module and a weak network decision module. Wherein, the decoding module of the receiving end D is used to decode the received audio and video stream (as shown in FIG. 7 , audio and video stream 1 & audio and video stream 2 & audio and video stream 3). As an example, the decoding module may include multiple decoders, and the multiple decoders are used to share the decoding work of audio and video streams. For example, the decoding module includes a decoder A, a decoder B and a decoder C, the decoder A is used to decode the audio and video stream 1 , the decoder B is used to decode the audio and video stream 2 , and the decoder C is used to decode the audio and video stream 3 .

The weak network decision-making module shown in Figure 7 is used to carry out overall bandwidth prediction (i.e. bandwidth resource Prediction) to get the total bandwidth prediction value. Further, the weak network decision-making module shown in Figure 7 is also used to make weak network judgments based on the total bandwidth prediction value and the bandwidth requirements of audio and video streams, and when it is determined to be in a weak network environment, the video in the window with a lower priority is preferentially downgraded and subscribed , to ensure the smooth playback of videos in windows with higher priority.

Exemplarily, the weak network decision-making module may downgrade the subscription to the video in the lower priority window, which may include but not limited to one or more of the following: unsubscribe, resume subscription, delay subscription, reduce definition, improve clarity degree etc.

It should be noted that the above FIG. 7 only uses the third device as an SFU as an example, and this embodiment of the present application does not limit the specific structure, function, form, etc. of the third device. For example, the third device may also be an electronic device such as a smart phone. For example, the third device can form a peer-to-peer (P2P) network architecture with the first device and the second device, wherein the first device, the second device and the third device can all serve as the receiver. As an example, in a multi-window video communication scenario, the third device may also serve as a forwarding device, forwarding the audio and video stream from the first device to the second device.

The following will take the multi-window video communication architecture shown in Figure 7 as an example, that is, multiple sending ends (that is, the first device) send audio and video streams to the receiving end (that is, the second device) through the third device (such as SFU) as an example , taking the display interface of the receiving end including the large window, small window 1, and small window 2 as shown in FIG. 4 as an example, a multi-window video communication method provided by the embodiment of the present application is specifically introduced with reference to the accompanying drawings.

As shown in Figure 8, the multi-window video communication method provided by the embodiment of the present application may include the following steps S801-S804:

S801. Multiple sending ends (that is, the first device) send audio and video streams to a third device (such as the SFU).

Wherein, in addition to including audio and video information, the audio and video stream also carries the identification (identification, ID) of the receiving end (such as the second device), which is used for the third device (such as SFU) to forward the audio to the receiving end according to the ID of the receiving end. video stream. Further, in some embodiments, the ID of the receiver carried in the audio and video stream is also used by the SFU to predict the downlink bandwidth of the corresponding downlink path.

In the embodiment of the present application, the audio and video stream also carries the ID of the sending end.

Taking the multi-window video communication architecture shown in Figure 7 as an example, as shown in Figure 9, the above step S801 may specifically include: the sending end A sends the audio and video stream 1 to the RTC SFU 1, and the sending end B and the sending end C send the audio and video stream 1 to the RTC SFU 1 respectively. 2 Send audio and video stream 2 and audio and video stream 3. Among them, the audio and video stream 1, the audio and video stream 2 and the audio and video stream 3 carry the ID of the sending end D. Further, the audio and video stream 1 carries the ID of the sender A, the audio and video stream 2 carries the ID of the sender B, and the audio and video stream 3 carries the ID of the sender C.

As an example, the audio and video streams in this embodiment of the present application may also carry the following information: stream resolution information, frame rate information, encoding Codec, stream level, and the like.

Exemplarily, audio-video stream 1, audio-video stream 2, and audio-video stream 3 shown in FIG. 9 not only include audio-video information, but also carry information as shown in Table 4 below:

Table 4

It should be noted that, the embodiment of the present application does not limit the specific order and timing of sending audio and video streams by multiple sending ends. For example, the sending end A, sending end B, and sending end C shown in FIG. 7 may send audio and video streams at the same time, or may send audio and video streams in any order of time.

S802. The third device (such as the SFU) forwards audio and video streams to the receiving end (such as the second device), predicts downlink bandwidth, and sends the bandwidth prediction result to the receiving end.

Taking the multi-window video communication architecture shown in FIG. 7 as an example, as shown in FIG. 9, the above step S802 may specifically include: the RTC SFU 1 transfers the voice video stream 1 to the receiving end D, and simultaneously performs bandwidth prediction to obtain a bandwidth prediction value 1, and Send the bandwidth prediction value 1 to the receiving end D; RTC SFU 2 transfers the voice video stream 2 to the receiving end D, and at the same time performs bandwidth prediction to obtain the bandwidth prediction value 2, and sends the bandwidth prediction value 2 to the receiving end D; RTC SFU 2 to the receiving end D forwards the audio and video stream 3, performs bandwidth prediction at the same time to obtain a bandwidth prediction value 3, and sends the bandwidth prediction value 3 to the receiving end D. For example, the predicted bandwidth value 1, the predicted bandwidth value 2, and the predicted bandwidth value 3 shown in FIG. 9 are all 500 kbps.

Exemplarily, the third device (such as the SFU) may forward the audio-video stream to the receiver according to the ID of the receiver carried in the audio-video stream.

For example, RTC SFU 1 shown in Figure 9 can forward audio and video stream 1 to receiver D according to the ID of receiver D carried in audio and video stream 1 from transmitter A; RTC SFU 2 shown in Figure 9 can transmit audio and video stream 1 based on The ID of the receiving end D carried in the audio and video stream 2 of the terminal B forwards the audio video stream 2 to the receiving end D; and the RTC SFU 2 transmits the ID of the receiving end D to the The receiving end D forwards the audio and video stream 3.

Further, in some embodiments, the third device (such as the SFU) can predict the downlink bandwidth of the corresponding downlink path according to the ID of the receiver carried in the audio and video stream.

Wherein, in the embodiment of the present application, the audio and video corresponding to the audio and video streams forwarded by multiple sending ends to the receiving end through a third device (such as SFU) are respectively played in multiple windows displayed on the display screen of the receiving end.

Please refer to FIG. 10 , which shows several examples of multi-window display. Among them, (a) in FIG. 10 and (b) in FIG. 10 show the multi-window display interface of the receiving end in a multi-window video communication scenario (such as a multi-party conference scenario or a group video chat scenario). (c) in FIG. 10 shows the multi-window display interface of the receiving end in the online education scenario.

It should be noted that FIG. 10 only shows windows related to this application. In some examples, function keys, menu bars, navigation keys/bars, etc. may also be displayed on the interface of the receiving end, which is not limited by this application.

Taking the display interface of the receiving end D shown in Figure 7 including the large window, the small window 1 and the small window 2 as shown in Figure 4 as an example, the sending end A, the sending end B, and the sending end C shown in Figure 7 respectively report to the receiving end D The sent audio-video stream 1, audio-video stream 2, and audio-video stream 3 are used to play in the large window, small window 1, and small window 2 of the receiving end D respectively.

Further, in some embodiments, the SFU may also send the bandwidth requirement to the receiving end. For example, the required bandwidths of audio-video stream 1, audio-video stream 2, and audio-video stream 3 shown in FIG. 9 are 600 kbps, 500 kbps, and 500 kbps respectively.

Further, in some embodiments, for example, for a video encoding method in which frames can be extracted, the third device (such as SFU) may also send the frame extraction status and the corresponding bandwidth requirement to the receiving end. Wherein, the frame extraction state is used to indicate whether the video coding mode is to extract frames or not to extract frames. For example, the frame extraction state information and corresponding bandwidth requirements of audio-video stream 1, audio-video stream 2, and audio-video stream 3 shown in FIG. 9 are shown in Table 5 below:

table 5

音视频流audio and video streaming	抽帧状态Frame state	带宽需求bandwidth requirements
音视频流1Audio and video stream 1	不抽帧no frames	400kbps400kbps
音视频流2Audio and video stream 2	抽帧Frame	300kbps300kbps
音视频流3Audio and video stream 3	抽帧Frame	300kbps300kbps

S803. The receiving end obtains a total bandwidth prediction value according to bandwidth prediction results from one or more third devices (such as SFU).

Wherein, the size of the total bandwidth prediction value can be used to represent the amount of bandwidth resources. For example, a larger total bandwidth prediction value indicates more sufficient bandwidth resources; a smaller total bandwidth prediction value indicates less bandwidth resources.

Taking the multi-window video communication architecture shown in FIG. 7 as an example, as shown in FIG. 9, the above step S803 may specifically include: the receiving end D according to the bandwidth prediction value 1 from the RTC SFU 1, the bandwidth prediction value 2 from the RTC SFU 2 and The bandwidth prediction value is 3, and the total bandwidth prediction value is obtained.

As an example, total bandwidth prediction value=bandwidth prediction value 1+bandwidth prediction value 2+bandwidth prediction value 3. For example, assuming that the predicted bandwidth value 1, the predicted bandwidth value 2, and the predicted bandwidth value 3 shown in FIG. 9 are all 500 kbps, it can be obtained that the total predicted bandwidth value is 1500 kbps.

It should be noted that this application does not limit the specific algorithm for the receiving end to obtain the total bandwidth prediction value based on the bandwidth prediction results from one or more SFUs. For this part, reference can be made to calculation methods in conventional technologies.

S804. When determining a weak network according to the total bandwidth prediction value, the receiving end adjusts a subscription strategy for audio and video streams in one or more windows according to priorities corresponding to the multiple windows.

Taking the multi-window video communication architecture shown in FIG. 7 as an example, as shown in FIG. 9, the above step S804 may specifically include: when determining a weak network according to the total bandwidth prediction value, the receiving end D according to the large window, small window 1 and small window 2, adjust the subscription strategy for audio and video streams in one or more windows of the large window, small window 1, and small window 2.

It can be understood that, in this embodiment of the present application, in order to give priority to ensuring smoothness and/or clarity of playing important videos, multiple windows on the display screen of the receiving end may have priority attributes.

Wherein, the priority corresponding to the window is used to represent the importance of the video played in the window to the user at the receiving end. For example, relatively important windows have a higher priority than relatively less important windows.

For example, on the display interface of the multi-window video communication scene shown in Figure 11, the large window plays high-definition video at a higher resolution (540P), and the

small windows

1 and 2 play normal-definition video at a lower resolution (360P). It can be understood that the video played in the large window is more important to the user than the videos played in the small window 1 and the small window 2. Therefore, the priority of the large window shown in FIG. 11 is higher than that of the small window 1 and the small window 2 .

It should be noted that the above examples in this application only take the resolution corresponding to HD video as 540P and the resolution corresponding to normal HD video as 360P as an example, and this application does not limit the specific definition of videos with different resolutions.

As another example, on the display interface of the multi-window video communication scene shown in (c) in FIG. 10 , the courseware/whiteboard/screen sharing window is the core of online education, while the portrait of the lecturer is relatively unimportant. Therefore, the priority of the courseware/whiteboard/screen sharing window is higher than that of the lecturer portrait window. The specific method for determining the priority will be introduced below.

Wherein, in the embodiment of the present application, weak network means that under the current network environment, bandwidth resources are insufficient to meet the bandwidth requirements of audio and video streams. Exemplarily, the weak network means that under the current network environment, the total bandwidth prediction value obtained by the receiving end (for example, 300kbps-1000kbps) is smaller than the bandwidth value required by the audio and video stream (for example, 1200kbps).

It can be understood that compared with video streams, audio streams require less communication resources. Therefore, in the embodiment of the present application, it is also possible to determine whether the network is in a weak network environment based only on the bandwidth requirements of video streams.

Taking the determination of whether it is in a weak network environment based on the bandwidth requirements of the video stream as an example, in some embodiments, for example, for the video encoding method without frame extraction, the weak network means that under the current network environment, the bandwidth resources are insufficient to meet the video stream frame extraction. previous bandwidth requirements.

For example, the video stream corresponding to the large window displayed on the receiving end D requires a bandwidth of 700 kbps before frame extraction, and the video stream corresponding to small window 1 and window 2 requires a bandwidth of 500 kbps before frame extraction. The total bandwidth value of is 700kbps+500kbps+500kbps=1700kbps. Assuming that the total bandwidth prediction value is 1500kbps, since the total bandwidth prediction value of 1500kbps is less than the total bandwidth value required by the video stream, the receiving end can determine that it is currently in a weak network environment.

In some other embodiments, for example, for the video encoding method of frame extraction, a weak network means that under the current network environment, bandwidth resources are insufficient to meet the bandwidth requirement of the video stream after frame extraction.

For example, the video stream corresponding to the large window displayed on the receiving end D requires a bandwidth of 400 kbps after frame extraction, and the video stream corresponding to small window 1 and small window 2 requires a bandwidth of 300 kbps before frame extraction. The total bandwidth value of is 400kbps+300kbps+300kbps=1000kbps. Assuming that the total bandwidth prediction value is 900kbps, since the total bandwidth prediction value of 900kbps is smaller than the total bandwidth value required by the video stream, the receiving end can determine that it is in a weak network environment.

It should be noted that the embodiment of the present application uses layered coding as an example, and for other video coding methods, the multi-window video communication method provided in the embodiment of the present application is also applicable. And, the embodiment of the present application takes the layered encoding of IDR frame+P frame+B frame, or IDR frame+P frame as an example. For other layered encoding types, the multi-window video communication method provided by the embodiment of the present application is also applicable .

As an example, in this embodiment of the application, the adjustment of the subscription policy by the receiving end may include, but not limited to, one or more of the following: unsubscribing, resuming subscription, delaying subscription, reducing clarity, improving clarity, and the like.

In some embodiments, when the receiving end adjusts the subscription policy, it can only adjust the subscription policy for the video stream, and keep the audio stream playing normally, so as to ensure the normal voice communication and communication of the user and ensure the user experience.

Take the subscription policy adjustment of the video stream as an example, where unsubscribing refers to unsubscribing from the corresponding video stream to cancel the video display in the window. Restoring the subscription refers to resuming the subscription to the corresponding video stream to restore the video display in the window. Delayed subscription refers to delayed subscription to the corresponding video stream, and the video display in the delayed window. Reducing the definition refers to subscribing to the reduced-definition video stream for the window, for example, switching from subscribing to a high-definition video stream to subscribing to a normal-definition video stream. Raising the definition refers to subscribing to the video stream with improved definition for the window, for example, switching from subscribing to a normal-definition video stream to subscribing to a high-definition video stream.

In the following, the display interface of the receiving end includes m windows (m≥3, m is an integer), and all m windows display high-definition video as an example. The specific examples illustrate that the receiving end cancels subscription, resumes subscription, The process of delaying subscription, reducing clarity, or increasing clarity. It should be noted that the embodiment of the present application does not limit the number of windows on the display interface of the receiving end, for example, the display interface of the receiving end may further include 2 windows.

(1), reduce clarity

Exemplarily, it is assumed that the display interface of the receiving end includes windows W ₁ , W ₂ ... W _m (where m represents the priority corresponding to the window), among the windows W ₁ , W ₂ ... W _m , the priority corresponding to W ₁ is the highest , the priority corresponding to W ₂ is the second, and the priority corresponding to W _m is the lowest. If the bandwidth resources are not enough to meet the bandwidth requirements of keeping windows W ₁ , W ₂ ... W _m at the current resolution, but meeting the requirements of keeping window W ₁ , W ₂ ... W _m-1 are displayed with the current parameters and the window W _m lowers the resolution display bandwidth requirement, then the receiving end decides to subscribe to the lowered resolution video stream for the window W _m , so as to ensure that the high-priority window (such as The resolution of the video in windows W ₁ , W _{2 .} . . W _m-1 ). For example, the receiving end decides to subscribe to the audio and video stream of the first definition for the window, and reduce the subscription to the audio and video stream of the second definition for the window. Wherein, the second definition is smaller than the first definition.

For example, assuming that the current windows W ₁ , W ₂ ... W _m all display high-definition video (that is, the first-definition video), RW _1/HD +RW _2/HD +...+RW _m/HD > predicted total bandwidth ≥RW _1/HD +RW _2/HD +…+RW _m/Pu , then the receiving end decides to reduce the definition of the video in the window W _m to ensure the video in the window W ₁ , W ₂ ……W _m-1 HD display. That is, the receiving end decides to subscribe to the normal-definition video stream (that is, the second-definition video stream) for the window W _m . Wherein, RW ₁ , RW ₂ . . . RW _m are bandwidth requirements of windows W ₁ , W ₂ . . . W _m respectively. Exemplarily, the resolution of high-definition video may be 540P, and the resolution of normal-definition video may be 360P.

In some embodiments, after the receiving end reduces the definition of the video in the window W _m , if RW _1/HD +RW _2/HD +...+RW _m-1/HD +RW _m/plain > total bandwidth prediction value ≥ _{RW 1/HD} +RW _2/HD +...+RW _m-1/Pudio +RW _m/Pu , then the receiving end can further decide to subscribe to PQ video stream for window W _m-1 (that is, the second high-definition video stream), thereby reducing the definition of the video in the window W _m-1 to ensure the high-definition display of the video in the windows W ₁ , W ₂ . . . W _m-2 , and so on.

In some other embodiments, after the receiving end reduces the video definition in the window W _m , if RW _1/HD +RW _2/ _HD +...+RW _m-1/HD +RW _m/Pu >total bandwidth Predicted value ≥ _{RW 1/HD} +RW _2/HD +…+RW _{m-1/Pu Qing} +RW _m/Pu _Qing , the receiving end can further decide to cancel the subscription of video stream for window W _m , thereby canceling window W _m The video in the window is displayed to ensure the high-definition display of the video in the windows W ₁ , W ₂ . . . W _m-2 .

It should be noted that the above examples only take the definition as an example including high-definition and normal definition, and this application does not limit the definition rules. For example, the definition may also include three levels of ultra-clear, high-definition and normal definition. For this situation, in some embodiments, when reducing the definition, it can be reduced sequentially according to the gradient of ultra-definition→high-definition→normal definition.

(2), unsubscribe

Exemplarily _, it is assumed _that _the display interface of the receiving end includes windows W ₁ , W ₂ _. If the bandwidth requirements for high-definition display are satisfied, but the bandwidth requirements for keeping windows W ₁ , W ₂ ... W _m-1 displayed with the current parameters and the window W _m does not display video are met, the receiving end decides to cancel the window W _m with the lowest priority Subscribe to the video stream, thereby canceling the display of the video in the lowest priority window to ensure that other windows are displayed in the first definition. .

For example, the receiving end decides to cancel the subscription of the video stream for the window instead of subscribing to the second-definition audio and video stream for the window with the lowest priority (that is, to cancel the subscription of the video stream for the window). Wherein, the second definition is less than or equal to a preset value.

For example, assuming that the current windows W ₁ , W ₂ ... W _m-1 all display high-definition video, and window W _m displays normal-definition video (that is, the second-definition video), if RW _1/HD +RW _2/HD +... …+RW _{m normal clear} > total bandwidth prediction value, then the receiving end decides to cancel the video stream subscription for window W _m , thereby canceling the video display in window W _m , so as to ensure that the windows W ₁ , W ₂ ... W _m-1 HD display of video. In some embodiments, after the receiving end cancels the video stream subscription for the window W _m , if RW _1/HD +RW _2/HD +...+RW _m-1/HD >total bandwidth prediction value≥RW _1/HD + RW _2/HD

+...+RW _{m-1/common definition} , then the receiver can further decide to subscribe to the video stream with reduced definition (such as normal definition) for window W _m-1 , thereby reducing the definition of the video in window W _m-1 , or, the receiving end can further decide to cancel the subscription of the video stream for the window W _m-1 to ensure the high-definition display of the video in the windows W ₁ , W ₂ ... W _m-2 , and so on.

As an implementation manner, in this embodiment of the application, the window for unsubscribing may display the last image frame before unsubscribing.

As another implementation manner, in the embodiment of the present application, the unsubscribe window may display a mask. For example, FIG. 12 shows an example of displaying a mask in the window after the widget 2 with a lower priority (priority is 2) unsubscribes in a weak network environment.

As another implementation manner, in the embodiment of the present application, the unsubscribed window may display a mask superimposed on the last frame image before unsubscribed.

It should be noted that, in the embodiment of the present application, in order to maintain the bandwidth prediction of the third device (such as SFU) for downlink transmission, at least one of the windows W ₁ , W ₂ . . . _Wm must be guaranteed to display video in at least one window. For example, at least ensure that the video in window _W1 is displayed at the lowest resolution.

It should be noted that this application does not limit the specific adjustment strategy adopted by the receiver device for subscribing to audio and video streams when it is in a weak network environment. When the bandwidth resources are insufficient to meet the bandwidth requirements of all windows to keep the video stream displayed in the first definition, but the video display in the window with the lowest priority is canceled and other windows display the required bandwidth in the first definition, then the receiving end It may also be decided to cancel the subscription of the video stream for the window with the lowest priority, thereby canceling the display of the video in the window with the lowest priority, so as to ensure that other windows are displayed with the first definition.

For example, assuming that the current windows W ₁ , W ₂ ... W _m all display high-definition video, if RW _1/HD +RW _2/HD

+...+RW _m/HD >total bandwidth prediction value, then the receiving end decides to cancel the video stream subscription for window W _m , thereby canceling the video display in window W _m , so as to ensure that windows W ₁ , W ₂ ... W _m- HD display of videos in ₁ .

(3), resume subscription

In some embodiments, if the receiving end monitors the total bandwidth prediction value (ie, bandwidth resource monitoring), and determines that the latest total bandwidth prediction value satisfies the unsubscribed window, the window with the highest priority resumes subscription and other windows remain currently displayed bandwidth requirement (that is, the second preset condition), the receiving end decides to revert to the unsubscribed window, and the window with the highest priority subscribes to the video stream, so as to restore the display of the corresponding video.

In some other embodiments, if the receiver determines that the latest total bandwidth prediction value meets the unsubscribed window within a preset time (such as 6 seconds) through the total bandwidth prediction value monitoring (i.e., bandwidth resource monitoring), priority If the window with the highest priority resumes subscription and other windows maintain the current display bandwidth requirements, the receiving end decides to revert to the unsubscribed window, and the window with the highest priority subscribes to the video stream to resume the display of the corresponding video.

For example, assuming that the current windows W ₁ , W ₂ ... W _m-1 all display high-definition video (that is, the first-definition video), and the window W _m does not display video, if the receiving end is monitored by the total bandwidth prediction value, it is determined that the continuous 6 Within seconds, if the total bandwidth prediction value is greater _{than or equal to} _{RW 1/HD} +RW _2/HD +…+RW m, then the receiving end decides to revert to subscribing to the video stream for window W _m , so as to restore the video display in window W _m . For example, the receiving end decides to resume subscribing to the second-definition video for the window W _m .

As another example, assuming that the current windows W ₁ , W ₂ ... W _m-2 all display high-definition video (that is, the first-definition video), and windows W _m-1 and W _m do not display video, if the receiving end passes the total bandwidth prediction Value monitoring, confirm that within 6 consecutive seconds, RW _1/ _HD +RW _2/HD +…+RW _m-1PuD +RW _mPuD >total bandwidth prediction value≥RW _1/HD +RW _2/HD +… …+RW _{m-1 normal clear} , then the receiving end decides to revert to subscribing to the video stream of window W _m-1 to restore the video display in window W _m-1 , but the status of window W _m is still unsubscribed.

(4), delayed subscription

Delayed subscription means that when a window with a higher priority than a certain window is in the unsubscribed state, it delays subscribing to the video stream for the window, so as to delay the restoration of the video display in the window.

For example, assuming that the current windows W ₁ , W ₂ ... W _m-2 all display high-definition video, and windows W _m-1 and W _m do not display video, then before window W _m-1 resumes subscription, the state of window W _m Still unsubscribe.

(5), improve clarity

In some embodiments, after reducing the definition, if the receiving end is monitored by the total bandwidth prediction value (ie, bandwidth resource monitoring), and it is determined that the latest total bandwidth prediction value satisfies multiple windows that have been reduced in definition, the priority is the highest. The window improves the definition display and other windows maintain the current display bandwidth requirements (that is, the first preset condition), then the receiving end decides to improve the definition of the video in the window with the highest priority among multiple windows that have been reduced in definition , to ensure the clarity of the video in the high-priority window.

In some other embodiments, after reducing the definition, if the receiving end is monitored by the total bandwidth prediction value (i.e., bandwidth resource monitoring), it is determined that within a preset time (such as 6 seconds), the latest total bandwidth prediction value satisfies multiple Among the windows whose resolution has been reduced, the window with the highest priority is displayed with higher resolution and other windows maintain the bandwidth requirements of the current display, then the receiving end decides to increase the resolution of the windows with the highest priority The clarity of the video to ensure the clarity of the video in the high priority window.

For example _, assuming that the current windows W ₁ , W ₂ _. After monitoring the total bandwidth prediction value, it is determined that within 6 consecutive seconds, the total bandwidth prediction value ≥ _{RW 1/HD} +RW _2/HD +...+RW _{m-1 HD} , then the receiving end decides to increase the definition of the window W _m-1 , for example, switching from normal definition display (that is, display with the second definition) to high definition display (that is, display with the first definition).

_As _another example, assume that the current windows W ₁ , W ₂ _. video), if the receiving end has been monitored by the total bandwidth prediction value, it is determined that within 6 consecutive seconds, RW _1/HD +RW _2/HD +...+RW _{m-1 HD} +RW _mHD >total bandwidth prediction value ≥ _{RW 1/} _HD +RW _2/HD +...+RW _{m-1 HD} , then the receiving end decides to improve the definition of window W _m-1 , for example, switch from ordinary clear display (that is, display with the second definition) to high-definition display (that is, is displayed in the first definition), but the window _Wm still displays the normal definition video (that is, the video in the second definition).

It should be noted that the above (1)-(5) are only examples of several subscription policy adjustments. In some embodiments, the receiving end can adjust the subscription of audio and video streams in multiple windows according to the priorities corresponding to multiple windows. Strategy.

For example, the receiving end may decide to reduce the resolution of the video in the window W _m-1 while unsubscribing from the window W _m (for example, switching from the first resolution display to the second resolution display.)

For another example, the receiving end may decide to reduce the definition of the video in the window _Wm- 1 while reducing the definition of the video in the window Wm _-1 . For example, switching from the first definition display to the second definition display.

Similarly, when the network is getting better, the receiving end can decide to improve the definition of the video in the window Wm _-1 while restoring the display of the window _Wm in the second definition (for example, switching from the second definition display to the first definition show.)

For another example, when the network is improving, the receiving end may decide to increase the definition of the video in the window _Wm- 1 while improving the definition of the video in the window Wm _-1 . For example, switching from the second-definition display to the first-definition display.

Further, after the receiving end determines the subscription strategy for audio and video streams in one or more windows, as shown in Figure 13, the method provided by the embodiment of the present application further includes step S805:

S805. The receiving end subscribes to multiple sending ends for audio and video streams according to the latest subscription policy.

Taking the multi-window video communication architecture shown in Figure 7 as an example, as shown in Figure 9, the above step S801 may specifically include: the receiving end D subscribes to the sending end A, the sending end B, and the sending end C according to the determined latest subscription policy. flow.

For example, the receiving end may send the following information to the third device to request the third device to subscribe to the corresponding audio and video stream from the first device: the identification (ID) of the sending end, the priority corresponding to the window, the original subscription parameter, the target subscription parameter, the window Identification (Surface ID).

Exemplarily, it is assumed that the receiving end determines to subscribe to the high-definition video stream for the window W _m and switch to subscribing for the ordinary clear video stream for the window W _m , wherein the sending end of the audio and video stream corresponding to the window W _m is the sending end A, and the window W _m The corresponding priority is 3, then the receiving end can send the following information to the third device to request the third device to subscribe to the first device for the audio and video stream of the corresponding parameters: ID of sending end A, 3 (priority), high-definition ( original subscription parameter), common clear (target subscription parameter), and the ID of the window W _m .

It should be noted that the above-mentioned embodiments shown in FIG. 8 and FIG. 9 of this application only use the third device (such as SFU) to predict the downlink bandwidth and send the bandwidth prediction result to the receiving end as an example, and this application does not limit the specific device responsible for downlink bandwidth prediction. equipment. For example, in the embodiment of the present application, the receiving end (that is, the second device) may also measure and obtain bandwidth prediction results of multiple links during the process of receiving multiple audio and video streams. Wherein, the above multiple links correspond to multiple audio and video streams. For example, the above multiple links are respectively used to transmit the above multiple audio and video streams.

As another implementation, in the embodiment of this application, if there is a window that reduces the clarity, cancels subscription or delays subscription due to being in a weak network environment, a prompt message can be displayed on the display screen of the electronic device to remind the user that the current network is relatively weak. Difference. For example, FIG. 14 shows an example of displaying a weak network prompt after widget 2 with a lower priority (priority 2) unsubscribes when in a weak network environment.

In the embodiment of the present application, the priority corresponding to the window can be specified by the user, or can be determined by the electronic device itself. The following will introduce several methods of determining the priority corresponding to the window with reference to the accompanying drawings:

(a) The priority corresponding to the window is specified by the user.

Exemplarily, as an implementation manner, the user can customize the priority setting by changing the size of the window. For example, a small window is enlarged to a large window, so as to increase the priority corresponding to the window. As shown in Figure 15, the priorities of window A, window B, window C and window D shown in (a) in Figure 15 are all 2, in response to the user's operation 1401 of stretching window A into a large window, as shown As shown in (b) in 15, the electronic device displays a large window D in the middle of the screen, and the electronic device sets the priority corresponding to window D to 1.

As another implementation manner, the user can change the order of the windows by dragging the windows, so as to customize the priority settings. For example, drag a window to the middle of the screen to increase the corresponding priority of the window. As shown in Figure 16, the priorities of window B, window C and window D shown in (a) in Figure 16 are all 2, in response to the user's operation 1501 of dragging window D to the middle of the screen, as shown in Figure 16 As shown in (b), the electronic device displays window D in the middle of the screen, and the electronic device sets the priority corresponding to window D to 1.

It should be noted that the embodiment of the present application does not specify a specific manner and method for the user to specify the priority corresponding to the window. For example, the user can also set the priority of the window in the menu.

(b), the priority corresponding to the window is determined by the electronic device according to the volume of the audio corresponding to the multiple windows.

As an implementation manner, the electronic device may determine the priorities corresponding to the windows according to the initial volumes of the audios corresponding to the multiple windows. Wherein, the initial volume of the audio is used to represent the original volume of the audio stream when the electronic device receives the audio stream.

It can be understood that when multiple users conduct a multi-party real-time video call, the volume of the current speaker is usually relatively loud. For example, in a multi-party conference scenario, the volume of the speaker of the current conference is usually relatively loud. For another example, in a group video chat scene, the volume of the currently speaking user is usually relatively loud. Therefore, in the embodiment of the present application, the electronic device may adaptively adjust the priority corresponding to the windows according to the initial volume of the audio corresponding to the multiple windows.

As another implementation manner, the electronic device may determine the priority corresponding to the windows according to the playback volume of the audio corresponding to the multiple windows.

It can be understood that when multiple users are conducting a multi-party real-time video call, the user at the receiving end can individually set the playback volume of the audio corresponding to different windows according to their concerns and points of interest. For example, for the window that the user pays most attention to, the user can increase the playback volume, and for the window that the user does not pay attention to, the user can lower the playback volume. Based on this, in the embodiment of the present application, the electronic device may adaptively adjust the priority corresponding to the windows according to the playback volume of the audio corresponding to the multiple windows.

(c), the priority corresponding to the window is determined by the electronic device according to the functions of the services in the multiple windows.

Taking the online education scene shown in (c) in Figure 10 as an example, the interface shown in (c) in Figure 10 includes the courseware/whiteboard/screen sharing window and the window where the lecturer portrait is located, wherein the courseware/whiteboard/screen sharing window It is used to display the courseware/whiteboard or display the shared interface, and the window where the lecturer portrait is located is used to play the lecturer video in real time. It can be understood that the core function of online education lies in the display of classroom content. Whether the lecturer portrait is smooth and clear will not affect the acquisition of classroom content. Therefore, the corresponding priority of the courseware/whiteboard/screen sharing window is higher than that of the lecturer portrait window.

It should be noted that the above-mentioned electronic device determines the priority corresponding to the window according to the volume of the audio corresponding to the multiple windows or the function of the business in the multiple windows as only two examples. The embodiment of the present application determines the priority corresponding to the window for the electronic device The specific rules and methods are not limited. For example, the priorities corresponding to the windows may also be determined by the electronic device according to other factors such as attributes of videos in multiple windows.

In the method provided in the embodiment of the present application, multiple windows may correspond to different priorities under different business scenarios or different user requirements. Electronic devices can downgrade and subscribe to video streams based on specific priorities when the downlink bandwidth is limited or the bandwidth fluctuates greatly, such as reducing video resolution, unsubscribing video streams, or delaying subscribing to video streams, etc., to avoid network congestion and ensure Smoothness and/or clarity of video is a high priority.

Further, when the electronic device determines that the network situation has improved, it can restore the unsubscribed video (that is, restore the subscription) or restore the definition of the downgraded video (that is, improve the definition), so as to ensure maximum protection in multiple windows. The smoothness and/or clarity of video playback.

It should be understood that various schemes of the embodiments of the present application can be used in a reasonable combination, and the explanations or descriptions of various terms appearing in the embodiments can be referred to or interpreted in each embodiment, which is not limited.

It should also be understood that in various embodiments of the present application, the serial numbers of the above-mentioned processes do not mean the order of execution, and the order of execution of the processes should be determined by their functions and internal logic, and should not be implemented in this application. The implementation of the examples constitutes no limitation.

It can be understood that, in order to realize the functions of any one of the above-mentioned embodiments, the electronic device (such as the first device, the second device or the third device) includes corresponding hardware structures and/or software modules for performing various functions. Those skilled in the art should easily realize that the present application can be implemented in the form of hardware or a combination of hardware and computer software in combination with the units and algorithm steps of each example described in the embodiments disclosed herein. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

The embodiment of the present application can divide the functional modules of the electronic device (such as the first device, the second device or the third device), for example, each functional module can be divided corresponding to each function, or two or more functions can be integrated in one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. It should be noted that the division of modules in the embodiment of the present application is schematic, and is only a logical function division, and there may be other division methods in actual implementation.

For example, in the case of dividing various functional modules in an integrated manner, as shown in FIG. 17 , it is a structural block diagram of an electronic device provided in the embodiment of the present application. For example, the electronic device may be a first device, a second device or a third device. As shown in FIG. 17 , the electronic device may include a transceiver unit 1710 , a processing unit 1720 and a storage unit 1730 .

Wherein, when the electronic device is a second device, the transceiver unit 1710 is configured to support the second device to receive audio and video streams from multiple first devices. For example, the transceiver unit 1710 is configured to support the second device to receive audio and video streams from multiple first devices forwarded by the third device. In some embodiments, the transceiver unit 1710 may also be configured to support the second device to receive bandwidth prediction values corresponding to multiple audio and video streams from the third device. Further, the transceiver unit 1710 may also be used to support the second device to subscribe to the audio and video stream from the first device, and/or other processes related to the embodiment of the present application.

The processing unit 1720 is configured to support the second device to obtain a total bandwidth prediction value according to multiple bandwidth prediction results, determine that it is in a weak network environment according to the total bandwidth prediction value, and adjust a subscription strategy for audio and video streams in one or more windows. In some embodiments, the processing unit 1720 is further configured to support the second device to measure bandwidth prediction results corresponding to multiple audio and video streams, and/or other processes related to the embodiments of the present application.

The storage unit 1730 is used to store computer programs and implement processing data and/or processing results in the methods provided by the embodiments of the present application.

It should be noted that the transceiver unit 1710 may include a radio frequency circuit. Specifically, the electronic device can receive and send wireless signals through a radio frequency circuit. Typically, radio frequency circuitry includes, but is not limited to, an antenna, at least one amplifier, transceiver, coupler, low noise amplifier, duplexer, and the like. In addition, radio frequency circuits can also communicate with other devices through wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile Communications, General Packet Radio Service, Code Division Multiple Access, Wideband Code Division Multiple Access, Long Term Evolution, Email, Short Message Service, etc.

It should be understood that each module in the electronic device may be implemented in the form of software and/or hardware, which is not specifically limited. In other words, electronic equipment is presented in the form of functional modules. The "module" here may refer to an application-specific integrated circuit ASIC, a circuit, a processor and memory executing one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the above-mentioned functions.

In an optional manner, when software is used to implement data transmission, it may be implemented in whole or in part in the form of computer program products. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are realized in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, etc. , tape), optical media (such as digital video disk (digital video disk, DVD)), or semiconductor media (such as solid state disk (SSD)), etc.

The steps of the methods or algorithms described in conjunction with the embodiments of the present application may be implemented in hardware, or may be implemented in a manner in which a processor executes software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, mobile hard disk, CD-ROM or any other form of storage known in the art medium. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC. Alternatively, the ASIC may be located in the electronic device. Of course, the processor and the storage medium can also exist in the electronic device as discrete components.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

Claims

A communication system, characterized in that the communication system includes:

The multiple first devices are configured to: send multiple audio and video streams to the second device, and the audio and video corresponding to the multiple audio and video streams are respectively played in multiple windows on the interface of the second device;

The second device is configured to: determine that bandwidth resources are insufficient to meet the bandwidth requirements of the multiple audio and video streams;

Adjust the subscription strategy for audio and video streams in one or more windows according to the priorities corresponding to the multiple windows.
The communication system according to claim 1, further comprising: one or more third devices;

The one or more third devices are configured to: receive the multiple audio and video streams from the multiple first devices;

Forward the multiple audio and video streams to the second device.
The communication system according to claim 2, wherein the one or more third devices are further configured to: forward multiple audio and video streams from the multiple first devices to the second device In the process, the measurement obtains multiple bandwidth prediction results for multiple links, and the multiple links correspond to the multiple audio and video streams;

The second device is configured to: determine, according to the multiple bandwidth prediction results, that the bandwidth resource is insufficient to meet the bandwidth requirements of the multiple audio and video streams.
The communication system according to claim 1, wherein the second device is further configured to: measure and obtain multiple bandwidth prediction results for multiple links during the process of receiving the multiple audio and video streams, The multiple links correspond to the multiple audio and video streams;

The second device is configured to: determine, according to the multiple bandwidth prediction results, that the bandwidth resource is insufficient to meet the bandwidth requirements of the multiple audio and video streams.
The communication system according to any one of claims 1-4, wherein the second device is used for:

The second device reduces the definition of video streams corresponding to one or more windows and/or unsubscribes from the video streams corresponding to one or more windows according to the priorities corresponding to the multiple windows.
A multi-window video communication method, characterized in that the method comprises:

The second device receives multiple audio and video streams from multiple first devices, and the audio and video corresponding to the multiple audio and video streams are respectively played in multiple windows on the interface of the second device;

The second device determines that bandwidth resources are insufficient to meet the bandwidth requirements of the multiple audio and video streams;

The second device adjusts a subscription strategy for audio and video streams in one or more windows according to priorities corresponding to the multiple windows.
The method according to claim 6, wherein the second device receives multiple audio and video streams respectively from the multiple first devices, comprising:

The second device receives multiple audio and video streams respectively from the multiple first devices forwarded by the third device.
The method according to claim 7, wherein the method further comprises:

The second device receives multiple bandwidth prediction results for multiple links from the third device, and the multiple links correspond to the multiple audio and video streams;

The second device determines that the bandwidth resources are insufficient to meet the bandwidth requirements of the multiple audio and video streams, including:

The second device determines, according to the multiple bandwidth prediction results, that the bandwidth resource is insufficient to meet the bandwidth requirements of the multiple audio and video streams.
The method according to claim 6 or 7, characterized in that the method further comprises:

The second device measures and obtains bandwidth prediction results for multiple links, and the multiple links correspond to the multiple audio and video streams;

The second device determines that the bandwidth resources are insufficient to meet the bandwidth requirements of the multiple audio and video streams, including:

The second device determines, according to the multiple bandwidth prediction results, that the bandwidth resource is insufficient to meet the bandwidth requirements of the multiple audio and video streams.
The method according to any one of claims 6-9, wherein the first window plays the corresponding audio and video stream with the first definition, and the first window is the window with the lowest priority among the plurality of windows ;

The second device adjusts a subscription strategy for audio and video streams in one or more windows according to the priorities corresponding to the multiple windows, including:

The second device subscribes to an audio and video stream of a second definition for the first window, and the second definition is smaller than the first definition.
The method according to claim 10, characterized in that, after the second device subscribes to the audio and video stream of the second definition for the first window, the method further comprises:

When the first preset condition is met, the second device subscribes the first window to the audio and video stream of the first definition.
The method according to any one of claims 6-9, wherein the second window plays the audio and video stream at a second definition, the second definition is less than or equal to a preset value, and the second window It is the window with the lowest priority among the plurality of windows, and the second device adjusts a subscription strategy for audio and video streams in one or more windows according to the priorities corresponding to the plurality of windows, including:

The second device unsubscribes from the video stream corresponding to the second window.
The method according to claim 12, wherein after the second device unsubscribes from the video stream corresponding to the second window, the method further comprises:

The second device displays a mask on the second window.
The method according to claim 12 or 13, wherein after the second device unsubscribes from the video stream corresponding to the second window, the method further comprises:

When the second preset condition is met, the second device resumes subscribing to the video stream of the second definition for the second window.
The method according to any one of claims 6-14, wherein the priorities corresponding to the plurality of windows are determined by the second device according to one or more of the following:

The initial volume of the audio corresponding to the plurality of windows; the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream;

The playback volume of the audio corresponding to the plurality of windows;

Functions of services in the multiple windows.
The method according to any one of claims 6-14, wherein the priorities corresponding to the plurality of windows are determined by the second device according to user-defined specified operations.
The method according to any one of claims 7-16, characterized in that the third device is a Selective Forwarding Unit (SFU).
An electronic device, characterized in that the electronic device comprises:

memory for storing computer programs;

A processor, configured to execute the computer program, so that the electronic device implements the method according to any one of claims 6-17.
A computer-readable storage medium, characterized in that computer program code is stored on the computer-readable storage medium, and when the computer program code is executed by a processing circuit, the method according to any one of claims 6-17 is realized. method.
A system on a chip, characterized in that the system on chip includes a processing circuit and a storage medium, and computer program codes are stored in the storage medium; when the computer program code is executed by the processing circuit, claims 6-17 are implemented. any one of the methods described.