WO2023011408A1 - Procédé, dispositif et système de communication vidéo multifenêtre - Google Patents

Procédé, dispositif et système de communication vidéo multifenêtre Download PDF

Info

Publication number
WO2023011408A1
WO2023011408A1 PCT/CN2022/109423 CN2022109423W WO2023011408A1 WO 2023011408 A1 WO2023011408 A1 WO 2023011408A1 CN 2022109423 W CN2022109423 W CN 2022109423W WO 2023011408 A1 WO2023011408 A1 WO 2023011408A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio
video
window
bandwidth
windows
Prior art date
Application number
PCT/CN2022/109423
Other languages
English (en)
Chinese (zh)
Inventor
张帮明
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023011408A1 publication Critical patent/WO2023011408A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/443—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems

Definitions

  • the embodiments of the present application relate to the field of communication technologies, and in particular, to a multi-window video communication method, device, and system.
  • multi-window video communication is used in professional conference scenarios (such as multi-party conference scenarios), life scenarios (such as group video chat scenarios) or online education scenarios in the form of multi-party video calls.
  • the sending end can forward the audio and video streams to the receiving end through the cloud.
  • the cloud side forwards the audio and video streams from the sending end to the receiving end, and performs bandwidth prediction on the communication link for forwarding the audio and video streams.
  • the bandwidth predicted by the cloud side is used to make communication decisions (such as frame extraction decisions, etc.) when the network is poor.
  • the present application provides a multi-window video communication method, device and system, which can make full use of bandwidth resources to avoid network congestion during a multi-party video call, thereby ensuring smoothness and/or clarity of the video.
  • a multi-window video communication method is provided, the method is applied in the process of video calling between a plurality of first devices and a second device, the method includes: the second device receives a plurality of video messages from a plurality of first devices respectively Audio and video streams; wherein, the audio and video corresponding to the above multiple audio and video streams are respectively played in multiple windows on the interface of the second device; the second device determines that the bandwidth resources are not enough to meet the bandwidth requirements of multiple audio and video streams; the second The second device adjusts the subscription strategy for audio and video streams in one or more windows according to the priorities corresponding to the multiple windows.
  • the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.
  • the bandwidth resources at the receiving end are not enough to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams
  • the priority corresponding to the window can be used to represent specific business requirements or user preferences.
  • the above-mentioned second device receiving multiple audio and video streams respectively from multiple first devices includes: the second device receiving multiple audio and video streams respectively from multiple first devices forwarded by a third device Audio and video streaming.
  • the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.
  • the above method further includes: the above-mentioned second device receives multiple bandwidth prediction results for multiple links from the third device; the second device determines that the bandwidth resource is insufficient to meet the Bandwidth requirements for multiple audio and video streams.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the third device may perform bandwidth prediction.
  • the third device may perform bandwidth prediction when forwarding audio and video streams to the second device.
  • the above method further includes: the second device measures and obtains multiple bandwidth prediction results for multiple links; the second device determines that the bandwidth resource is insufficient to meet the above multiple Bandwidth requirements for audio and video streams.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.
  • the first window plays the corresponding audio and video stream at the first definition; the first window is among the above-mentioned multiple windows; the second device adjusts one or more
  • the subscription strategy for audio and video streams in a window includes: the second device subscribes to an audio and video stream with a second definition for the first window; and the second definition is smaller than the first definition.
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the above method further includes: when the first preset condition is met, the second device Subscribe for audio and video streams in first definition.
  • the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.
  • the above method further includes: when the first preset condition is met for a preset time period, the second device Subscribe to the first definition audio and video stream for the first window.
  • the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.
  • the second window plays audio and video streams with a second definition, the second definition is less than or equal to a preset value, and the second window is a window with the lowest priority among the plurality of windows;
  • the second device adjusts the subscription strategy for the audio and video streams in one or more windows according to the priorities corresponding to the multiple windows, including: the second device unsubscribes the video stream corresponding to the second window.
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the above method further includes: the second device displays a mask on the second window.
  • the second device displays a mask on the second window.
  • the above method further includes: when a second preset condition is met, the second device resumes subscribing to the second video stream for the second window.
  • the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the above method further includes: when the second preset condition is met for a preset time period, the second device is the second window Resume subscription to second definition video stream.
  • the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the priorities corresponding to the multiple windows are determined by the second device according to one or more of the following: the initial volume of the audio corresponding to the multiple windows; the playback volume of the audio corresponding to the multiple windows ; The function of business in multiple windows.
  • the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream.
  • various window priority settings can be supported.
  • the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the priorities corresponding to the foregoing multiple windows are determined by the second device according to user-defined specified operations.
  • various window priority settings can be supported. For example, it can be user-defined.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the above-mentioned second device receiving multiple audio and video streams from multiple first devices forwarded by the third device includes: the second device receives the first audio and video stream from the first cloud device ; The second device receives the second audio-video stream and the third audio-video stream from the second cloud device.
  • the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks.
  • a distributed cloud device that is, a third device
  • the foregoing third device is a selective forwarding unit (selective forwarding unit, SFU).
  • SFU selective forwarding unit
  • an electronic device such as a second device
  • the electronic device includes: a transceiver unit, configured to receive a plurality of audio and video streams from a plurality of first devices; wherein, the plurality of audio and video streams correspond to The audio and video are respectively played in multiple windows on the interface of the second device; the display unit is used to play the multiple audio and video streams through the multiple windows; the processing unit is used to determine that bandwidth resources are not enough to satisfy multiple audio The bandwidth requirement of the video stream; and adjust the subscription strategy for the audio and video stream in one or more windows according to the priorities corresponding to multiple windows.
  • the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.
  • the bandwidth resources of the receiving end are insufficient to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams
  • the priority corresponding to the window may be used to represent a specific service requirement or a user's preference.
  • the above-mentioned transceiving unit is specifically configured to: receive multiple audio and video streams respectively from multiple first devices forwarded by the third device.
  • the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.
  • the transceiver unit is further configured to: receive multiple bandwidth prediction results for multiple links from the third device; the processing unit is further configured to: determine that the bandwidth resource is insufficient according to the multiple bandwidth prediction results To meet the bandwidth requirements of multiple audio and video streams.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the third device may perform bandwidth prediction.
  • the third device may perform bandwidth prediction when forwarding audio and video streams to the second device.
  • the above processing unit is further configured to: measure and obtain multiple bandwidth prediction results for multiple links; and determine that bandwidth resources are insufficient to meet the above multiple audio and video frequency requirements according to the above multiple bandwidth prediction results.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.
  • the first window plays the corresponding audio and video stream at the first definition; the first window is among the above-mentioned multiple windows; the above-mentioned processing unit is specifically configured to: subscribe the first window to the second definition audio and video stream; the second definition is smaller than the first definition.
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the above-mentioned processing unit is further configured to, when the first preset condition is satisfied, subscribe the first window to the audio-video stream of the first definition.
  • the processing unit may determine whether the first preset condition is met by monitoring the predicted value of the total bandwidth.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.
  • the above-mentioned processing unit is further configured to: when the first preset condition is met for a preset time period, subscribe to the first window for the audio and video stream of the first definition.
  • the processing unit may monitor whether the first preset condition is met by monitoring the total bandwidth prediction value.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.
  • the second window plays audio and video streams with a second definition, the second definition is less than or equal to a preset value, and the second window is a window with the lowest priority among the plurality of windows;
  • the above processing unit is specifically configured to: unsubscribe from the video stream corresponding to the second window.
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the display unit is further configured to: display a mask on the second window.
  • the above-mentioned processing unit is further configured to: resume subscribing to the video stream of the second definition for the second window when the second preset condition is satisfied.
  • the processing unit may determine whether the second preset condition is satisfied by monitoring the total bandwidth prediction value.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the processing unit is further configured to: when the second preset condition is met for a preset time period, the processing unit resumes subscribing to the video stream of the second definition for the second window.
  • the processing unit may determine whether the second preset condition is satisfied by monitoring the total bandwidth prediction value.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the above processing unit is further configured to determine the priorities corresponding to multiple windows according to one or more of the following: initial volume of audio corresponding to multiple windows; playback of audio corresponding to multiple windows Volume; a function of business in multiple windows.
  • the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream.
  • various window priority settings can be supported.
  • the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the above processing unit is further configured to determine priorities corresponding to multiple windows according to user-defined specified operations.
  • various window priority settings can be supported. For example, may be user-defined.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the above-mentioned transceiving unit is specifically configured to: receive the first audio-video stream from the first cloud device; and receive the second audio-video stream and the third audio-video stream from the second cloud device.
  • the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks.
  • a distributed cloud device that is, a third device
  • the foregoing third device is an SFU.
  • an electronic device such as a second device
  • the electronic device includes: a memory for storing a computer program; a transceiver for receiving or sending a radio signal; a display for displaying an interface; a processor, It is used to execute the computer program, so that the electronic device receives multiple audio and video streams from multiple first devices through a transceiver; wherein, the audio and video corresponding to the multiple audio and video streams are respectively displayed on the interface of the second device Play in multiple windows; determine that the bandwidth resources are not enough to meet the bandwidth requirements of multiple audio and video streams; and adjust the subscription strategy for the audio and video streams in one or more windows according to the priorities corresponding to the multiple windows.
  • the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.
  • the bandwidth resources at the receiving end are insufficient to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams
  • the priority corresponding to the window may be used to represent a specific service requirement or a user's preference.
  • the foregoing transceiver is specifically configured to: receive multiple audio and video streams respectively from multiple first devices forwarded by the third device.
  • the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.
  • the transceiver is further configured to: receive multiple bandwidth prediction results for multiple links from the third device; the processor is further configured to: determine that bandwidth resources are insufficient according to the multiple bandwidth prediction results To meet the bandwidth requirements of multiple audio and video streams.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the third device may perform bandwidth prediction.
  • the third device may perform bandwidth prediction when forwarding audio and video streams to the second device.
  • the above-mentioned processor is further configured to: measure and obtain multiple bandwidth prediction results for multiple links; The bandwidth requirements of the stream.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.
  • the first window plays the corresponding audio and video stream at the first definition; the first window is among the plurality of windows; the processor is specifically configured to: subscribe the first window to the second definition audio and video stream; the second definition is smaller than the first definition.
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the above-mentioned processor is further configured to: when the first preset condition is satisfied, subscribe the first window to the audio-video stream of the first definition.
  • the processor may monitor the total bandwidth prediction value to determine whether the first preset condition is met.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.
  • the above-mentioned processor is further configured to: subscribe for the first window to an audio and video stream of the first definition when the first preset condition is met for a preset time period.
  • the processor may monitor the total bandwidth prediction value to determine whether the first preset condition is met.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.
  • the second window plays audio and video streams with a second definition, the second definition is less than or equal to a preset value, and the second window is a window with the lowest priority among the plurality of windows;
  • the processor above is specifically configured to: unsubscribe from the video stream corresponding to the second window.
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the display is further configured to display a mask on the second window.
  • the above-mentioned processor is further configured to: resume subscribing to the video stream of the second definition for the second window when the second preset condition is met.
  • the processor may monitor the total bandwidth prediction value to determine whether the second preset condition is met.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the processor is further configured to: when the second preset condition is met for a preset time period, the processing unit resumes subscribing to the video stream of the second definition for the second window.
  • the processor may monitor the total bandwidth prediction value to determine whether the second preset condition is met.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the above-mentioned processor is further configured to determine the priorities corresponding to multiple windows according to one or more of the following: the initial volume of the audio corresponding to the multiple windows; the playback of the audio corresponding to the multiple windows Volume; a function of business in multiple windows.
  • the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream.
  • various window priority settings can be supported.
  • the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the above-mentioned processor is further configured to: determine priorities corresponding to multiple windows according to user-defined specified operations.
  • various window priority settings can be supported. For example, it can be user-defined.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the above-mentioned transceiver is specifically configured to: receive the first audio-video stream from the first cloud device; and receive the second audio-video stream and the third audio-video stream from the second cloud device.
  • the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks.
  • a distributed cloud device that is, a third device
  • the foregoing third device is an SFU.
  • a multi-window video communication method which is applied in the process of a video call between multiple first devices and a second device in a communication system, and the method includes: multiple first devices send to the second device Audio and video streams; wherein, the audio and video corresponding to the above multiple audio and video streams are respectively played in multiple windows on the interface of the second device; the second device determines that the bandwidth resources are not enough to meet the bandwidth requirements of multiple audio and video streams; the second The second device adjusts the subscription strategy for audio and video streams in one or more windows according to the priorities corresponding to the multiple windows.
  • the adjustment of the subscription policy by the second device may include, but not limited to, one or more of unsubscribing, resuming subscription, delaying subscription, reducing clarity, or increasing clarity.
  • the bandwidth resources at the receiving end are insufficient to meet the bandwidth requirements of multiple audio and video streams (i.e. a weak network environment), for example, when the total bandwidth prediction value used to characterize the bandwidth resources is less than the number of audio and video streams
  • the priority corresponding to the window may be used to represent a specific service requirement or a user's preference.
  • the above-mentioned communication system further includes: one or more third devices, configured to receive the above-mentioned multiple audio and video streams from the above-mentioned multiple first devices, and forward the above-mentioned multiple audio and video streams to the second device video stream.
  • the multi-window video communication method provided in this application can be applied to a network architecture where a third device forwards audio and video streams, so as to improve the applicability of the method provided in this application and compatibility with different network architectures.
  • the third device is further configured to: measure and obtain multiple bandwidth prediction results for multiple links during the process of forwarding the multiple audio and video streams to the second device;
  • the device is specifically configured to: determine, according to multiple bandwidth prediction results, that bandwidth resources are insufficient to meet bandwidth requirements of multiple audio and video streams.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the third device may perform bandwidth prediction.
  • the third device may perform bandwidth prediction when forwarding audio and video streams to the second device.
  • the second device is further configured to: during the process of receiving the above multiple audio and video streams, perform measurement to obtain multiple bandwidth prediction results for multiple links; the second device is specifically configured to: It is determined according to the multiple bandwidth prediction results that the bandwidth resources are insufficient to meet the bandwidth requirements of the multiple audio and video streams.
  • the above multiple links correspond to the above multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • the second device can perform bandwidth prediction, and through this solution, the compatibility of the method provided by the present application with different network architectures can be improved.
  • the above-mentioned second device is further configured to: play the corresponding audio and video stream with the first definition through the first window; the first window is among the above-mentioned multiple windows; the second device
  • the corresponding priority adjustment of the subscription strategy for the audio and video streams in one or more windows includes: the second device subscribes to the audio and video streams of the second definition for the first window; the second definition is lower than the first definition.
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy to reduce the video definition for low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the second device is further configured to: when the first preset condition is met, Subscribe for audio and video streams in first definition.
  • the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for restoring the definition is met, the definition of the degraded video is restored, so as to ensure the smoothness and/or definition of the video to the greatest extent.
  • the second device is further configured to: when the first preset condition is met for a preset time period, Subscribe to the first definition audio and video stream for the first window.
  • the second device may determine whether the first preset condition is satisfied by monitoring the predicted value of the total bandwidth.
  • the first preset condition may be that the latest total bandwidth prediction value satisfies the first window and resumes playing the audio and video stream at the first definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when meeting the bandwidth requirements for restoring clarity, restore the definition of the degraded video to maximize the fluency and/or clarity of the video
  • the second device is further configured to: play audio and video streams with a second definition through a second window, the second definition is less than or equal to a preset value, and the second window is the above-mentioned multiple Among the windows, the window with the lowest priority; the second device adjusts the subscription strategy for the audio and video streams in one or more windows according to the priorities corresponding to the multiple windows, including: the second device unsubscribes the video stream corresponding to the second window .
  • the receiving end when the receiving end is in a weak network environment, it can adjust the subscription policy of unsubscribing to low-priority windows to avoid network congestion while ensuring the fluency and/or clarity of high-priority videos.
  • the second device is further configured to: display a mask on the second window.
  • the mask layer By displaying the mask layer, users can be reminded that they are currently in a weak network environment and user experience can be improved.
  • the second device is further configured to: resume subscribing to the second window for the second window when the second preset condition is met.
  • high-definition video streaming For example, the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the second device is further configured to: when the second preset condition is met for a preset time period, set Resume subscription to second definition video stream.
  • the second device may determine whether the second preset condition is met by monitoring the predicted value of the total bandwidth.
  • the second preset condition may be that the latest total bandwidth prediction value satisfies the second window and resumes playing the audio and video stream at the second definition.
  • the receiving end can adjust the subscription policy according to the real-time bandwidth situation. For example, when the bandwidth requirement for resuming subscription is met, the subscription to the audio and video stream corresponding to the unsubscribed window is resumed, so as to ensure the fluency and/or clarity of the video to the greatest extent.
  • the second device is further configured to determine the priorities corresponding to the multiple windows according to one or more of the following: the initial volume of the audio corresponding to the multiple windows; The playback volume of the audio corresponding to the window; the function of business in multiple windows.
  • the initial volume of the audio is used to represent the original volume of the audio stream when the second device receives the audio stream.
  • various window priority settings can be supported.
  • the second device may determine the priorities corresponding to the multiple windows according to the volume of the initial/playing audio corresponding to the multiple windows and/or the functions of the services in the multiple windows.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the above-mentioned second device is further configured to: determine priorities corresponding to multiple windows according to user-defined specified operations.
  • various window priority settings can be supported. For example, it can be user-defined.
  • the diversified window priority setting can facilitate diversified operations of the user and improve user experience.
  • the above-mentioned one or more third devices include a first cloud device and a second cloud device, wherein the first cloud device is used to forward the first audio and video stream to the second device, and the second cloud device The device is configured to forward the second audio-video stream and the third audio-video stream to the second device.
  • the multi-window video communication method provided by this application can be applied to a network architecture in which a distributed cloud device (that is, a third device) forwards audio and video streams, so as to improve the applicability of the method provided by this application and to communicate with different networks.
  • a distributed cloud device that is, a third device
  • the foregoing third device is an SFU.
  • a communication system in a fifth aspect, includes: a plurality of first devices and the electronic device in any possible implementation manner of the second aspect or the third aspect.
  • the above-mentioned multiple first devices send audio and video streams to the second device for playing in multiple windows on the interface of the second device respectively.
  • the communication system is used to implement the method in any possible implementation manner of the fourth aspect.
  • the foregoing communication system further includes: one or more third devices, configured to forward audio and video streams from the plurality of first devices to an electronic device.
  • a computer-readable storage medium is provided.
  • Computer program code is stored on the computer-readable storage medium.
  • the processor can realize any possible implementation of the first aspect. methods in methods.
  • a chip system in a seventh aspect, includes a processor and a memory, and computer program code is stored in the memory; when the computer program code is executed by the processor, the processor implements any one of the first aspect. method in one possible implementation.
  • the system-on-a-chip may consist of chips, or may include chips and other discrete devices.
  • a computer program product comprising computer instructions.
  • the computer instructions When the computer instructions are run on the computer, the computer is made to implement the method in any possible implementation manner of the first aspect.
  • Fig. 1 is an exemplary diagram of a selective forwarding unit (selective forwarding unit, SFU) forwarding scheme provided by the embodiment of the present application;
  • FIG. 2 is a video communication interaction diagram based on SFU frame extraction provided by an embodiment of the present application
  • FIG. 3 is a schematic diagram of a layered encoding provided by an embodiment of the present application.
  • FIG. 4 is an example diagram of a multi-window video communication scenario provided by an embodiment of the present application.
  • FIG. 5A is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • FIG. 5B is a software architecture diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a call service architecture based on real time communication (real time communication, RTC) provided by the embodiment of the present application;
  • FIG. 7 is a structure diagram of a multi-window video communication provided by an embodiment of the present application.
  • FIG. 8 is a flowchart of a multi-window video communication method provided by an embodiment of the present application.
  • FIG. 9 is an interaction diagram of a multi-window video communication method provided by an embodiment of the present application.
  • FIG. 10 is an example diagram of three types of multi-window display provided by the embodiment of the present application.
  • FIG. 11 is an example diagram of a multi-window display provided by the embodiment of the present application.
  • Fig. 12 is an example diagram of displaying a mask when a weak network is provided by the embodiment of the present application.
  • FIG. 13 is a flow chart of another multi-window video communication method provided by the embodiment of the present application.
  • FIG. 14 is an example diagram of displaying a weak network prompt when a weak network is provided by an embodiment of the present application.
  • FIG. 15 is an example diagram 1 of a method for setting a priority corresponding to a window provided in an embodiment of the present application
  • Figure 16 is an example of the setting method of the priority corresponding to the window provided in the embodiment of the present application Figure 2;
  • FIG. 17 is a structural block diagram of an electronic device provided by an embodiment of the present application.
  • first and second are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as “first” and “second” may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, “plurality” means two or more.
  • An embodiment of the present application provides a multi-window video communication method, which is applied to a process in which multiple users conduct a multi-party real-time video call.
  • a multi-window video communication method provided in this embodiment of the application can be applied to a multi-party conference scenario.
  • user A, user B, and user C join a conference for a multi-party real-time video call, where user A is the conference speaker, and user B and user C are conference participants.
  • the multi-window video communication method provided in the embodiment of the present application may be applied to a group video chat scene.
  • user A, user B, and user C join a group chat for a multi-party real-time video call, wherein during the group chat, user A, user B, and user C all speak freely.
  • the multi-window video communication method provided in the embodiment of the present application may be applied to an online education scenario.
  • user A, user B, and user C join an online class group for a multi-party real-time video call, where user A is a teacher, and user B and user C are students.
  • User A shows courseware/whiteboard to user B and user C while teaching and explaining.
  • the sending end (such as the first device) can send a message to the receiving end through a third device (such as a cloud device) (such as the second device) forward audio and video streams.
  • a third device such as a cloud device
  • the third device may complete the forwarding of the audio and video stream based on the publishing and subscription relationship of the audio and video stream.
  • the third device supports the end-to-end encryption feature, and does not need to analyze the audio and video streams at the sending end.
  • the sending end and the receiving end store keys (such as a public key and a private key), which are not known by the third device, so the third device cannot parse the user's audio and video streams, which is highly secure.
  • the sending end may forward the audio and video streams to the receiving end through a selective forwarding unit (selective forwarding unit, SFU).
  • SFU selective forwarding unit
  • multiple SFUs can be deployed in a distributed manner to improve network scalability.
  • FIG. 1 shows an example diagram of an SFU forwarding solution. As shown in Figure 1, device 1 and device 2 respectively select SFU 1 and SFU 2 to transmit audio and video streams to device 4, and device 3 selects SFU3 to send audio and video streams to device 4 and device 5.
  • the sending end can select the optimal SFU for audio and video stream forwarding.
  • the sending end may select the optimal SFU according to network conditions (such as operators, regions, etc.), which is not limited in this application.
  • the third device (such as a cloud device) can also perform bandwidth prediction on the downlink of the forwarded audio and video stream when forwarding, and send the prediction result to the receiving end for use by the receiving end in the network. Time-lapse communication decisions.
  • a third device (such as an SFU) can make frame extraction decisions when the network is poor.
  • frame extraction can reduce downlink bandwidth consumption, avoid network congestion, and ensure audio and video fluency and/or clarity in a weak network environment.
  • FIG. 2 takes the bandwidth prediction performed by the SFU through the bandwidth prediction module as an example, and shows a video communication interaction diagram based on SFU frame extraction.
  • the video communication process based on SFU frame extraction mainly includes the following five steps:
  • Step 1 The sending end collects, encodes and encrypts audio and video data
  • Step 2 The sender sends the frame data, frame type, etc. to the SFU;
  • Step 3 The forwarding module of the SFU forwards the frame data to the receiving end, and at the same time, the bandwidth prediction module of the SFU performs downlink bandwidth prediction;
  • Step 4 SFU makes a frame drawing decision based on the bandwidth prediction value
  • Step 5 After receiving the frame data, the receiving end performs decryption, decoding and rendering.
  • the bandwidth prediction performed by the bandwidth prediction module may include: the downlink bandwidth prediction value calculated by the bandwidth prediction module according to data such as time delay and packet loss of downlink data transmission.
  • data such as time delay and packet loss of downlink data transmission.
  • IDR frame compression can be achieved 6:1 without any perceptible blurring. Using P frame compression at the same time as IDR frame compression can achieve a higher compression ratio without noticeable blurring.
  • B frame compression can achieve a compression ratio of 200:1, and its file size is generally 15% of the IDR frame compression size, less than half of the P frame compression size.
  • the IDR frame compression can remove the spatial redundancy of the image
  • the P frame and B frame compression can remove the temporal redundancy.
  • the IDR frame compression adopts full-frame compression encoding, that is, the full-frame image information is subjected to joint photographic experts group (JPEG) compression encoding.
  • JPEG joint photographic experts group
  • the IDR frame describes the details of the image background and moving subject, therefore, the IDR frame is also called a key frame. Based on this, IDR frames can be decoded and rendered independently. When decoding a video stream, only the data of the IDR frame can be used to reconstruct the complete picture.
  • the IDR frame can be used as a reference for P and B frames.
  • an IDR frame can be used as a reference for a P frame and a B frame after performing intra prediction, residual determination, residual transformation and quantization, variable length coding and arithmetic coding, image reconstruction and filtering, respectively.
  • the residual can be determined by subtracting the predicted value from the pixel value.
  • the P frame is a coded frame separated by 1 to 2 frames behind the IDR frame.
  • the P frame belongs to the interframe coding of forward prediction, so it only refers to the IDR frame or P frame closest to it for prediction.
  • the P frame adopts the method of motion compensation to predict the difference and the motion vector between the current frame and the previous nearest IDR frame or P frame.
  • the complete P frame image must be reconstructed after summing the prediction value and the prediction error in the IDR frame.
  • a P frame is predicted based on its preceding IDR frame.
  • the P frame can be the reference frame of the P frame behind it, or the reference frame of the B frame before and after it.
  • B-frames are bidirectionally inter-coded.
  • B frames extract data from preceding and following IDR frames or P frames.
  • the B frame is compressed based on the difference between the current frame and the image of the previous frame and the next frame to complete decoding and rendering. For example, the B frame predicts the prediction error and motion vector between the current frame and the preceding IDR frame or between the P frame and the following P frame. As shown in Figure 3, a B frame is predicted based on its preceding IDR frame and P frame, or based on its preceding and following P frames. B frames are not referenced by other frames.
  • B frames are not referenced by other frames, so the discarding of B frames will not affect the reference relationship of video frames. Based on this, when the network is poor at the receiving end, it can decide whether to extract B frames according to the bandwidth prediction value and the bandwidth requirements of various types of encoded frames. Among them, the elimination of B frames can achieve the purpose of reducing the downlink bandwidth load.
  • the bandwidth requirements of various types of coded frames can be obtained by counting the original frame data from the sending end.
  • Table 1 shows an example of bandwidth requirements.
  • the high-definition video is played in the large window, and the resolution of the video in the large window is 540P; the normal definition video is played in the small window, and the resolution of the video in the small window is 360P.
  • the required bandwidth of the video stream played in the large window shown in Figure 4 is 700 kilobits per second (kilobits per second, kbps) before drawing frames; the required bandwidth after drawing B frames is 400kbps.
  • the video streams played in the small window 1 and small window 2 shown in Figure 4 require a bandwidth of 500 kbps before frame extraction; and require a bandwidth of 300 kbps after extracting B frames.
  • the bandwidth required for audio and video streams is affected by the codec (Codec), the code rate control of the upstream bandwidth, the number of layers of layered coding, and the resolution/frame rate of the video.
  • codec codec
  • This example does not limit the specific calculation strategies and specific algorithms for the required bandwidth of audio and video streams.
  • the receiving end makes a decision Does not pump frames.
  • the receiving end decides to draw B frames.
  • the predicted bandwidth value is less than the required bandwidth of the IDR frame + the required bandwidth of the P frame, even if the B frame is extracted, the predicted bandwidth still cannot meet the required bandwidth after the frame is drawn. congestion. Network congestion will increase the delay, resulting in audio and video freezes.
  • the receiving end decides not to draw frames.
  • the bandwidth prediction value is less than 1000kbps, the predicted bandwidth cannot meet the minimum required bandwidth of the three windows. In this case, even if B frames are extracted, the predicted bandwidth still cannot meet the required bandwidth after frame extraction. Therefore, after frame extraction There are still audio and video freezes in the large window, small window 1, and small window 2 shown in Figure 4.
  • the receiver decides to draw B frames for the large window, small window 1, and small window 2 shown in Figure 4, so the large window, small window 1, and small window 2 The smoothness of the video in the medium and low-level videos has been reduced.
  • the predicted bandwidth value is less than 1000kbps
  • the receiver decides to draw B frames for the large window, small window 1, and small window 2 shown in Figure 4, but the predicted bandwidth after frame extraction still cannot meet the minimum requirements of the three windows. Bandwidth is required, so there is network congestion in all three windows. Network congestion will increase the delay, resulting in audio and video freezes in the large window, small window 1, and small window 2 shown in Figure 4.
  • an embodiment of the present application provides a multi-window video communication method.
  • the terminal data stream can be adjusted according to the specific business scenarios and demands of multiple video call users to ensure priority Normal playback of high priority video streams. For example, in the scenario shown in FIG. 4 , priority is given to ensuring video fluency and/or clarity in the large window. As another example, in online education scenarios, priority is given to ensuring the video fluency and/or clarity of courseware/whiteboards.
  • a weak network means that bandwidth resources are insufficient to meet the bandwidth requirements of audio and video streams.
  • the bandwidth is limited Limiting scenarios, etc.
  • using the method provided by the embodiment of this application to downgrade and subscribe to video streams based on specific priorities can refine the gradient of video stream control, and avoid network congestion while making full use of downlink bandwidth, thereby ensuring maximum protection The smoothness and/or clarity of the video.
  • the multi-window video communication method provided by the embodiment of the present application can be applied to, but not limited to, smart phones, netbooks, tablet computers, smart watches, smart bracelets, phone watches, smart cameras, palmtop computers, personal computers (personal computers, PCs), personal Digital assistant (personal digital assistant, PDA), portable multimedia player (portable multimedia player, PMP), augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) equipment, TV, projection equipment or human-computer interaction Somatosensory game consoles in the scene, etc.
  • the method may also be applied to electronic devices of other types or structures, which is not limited in this application.
  • FIG. 5A shows a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application by taking a smart phone as an example.
  • the electronic device can include a processor 510, a memory (comprising an external memory interface 520 and an internal memory 521), a universal serial bus (universal serial bus, USB) interface 530, a charging management module 540, and a power management module 541 , battery 542, antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, audio module 570, speaker 570A, receiver 570B, microphone 570C, earphone jack 570D, sensor module 580, button 590, motor 591, indicator 592 , a camera 593, a display screen 594, and a subscriber identification module (subscriber identification module, SIM) card interface 595, etc.
  • SIM subscriber identification module
  • the sensor module 580 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.
  • the structure shown in the embodiment of the present invention does not constitute a specific limitation on the electronic device.
  • the electronic device may include more or fewer components than shown in the illustrations, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • Processor 510 may include one or more processing units.
  • the processor 510 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a flight controller, Video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc.
  • application processor application processor
  • AP application processor
  • modem processor graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • flight controller Video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • a memory may also be provided in the processor 510 for storing instructions and data.
  • the memory in processor 510 is a cache memory.
  • the memory may hold instructions or data that the processor 510 has just used or recycled. If the processor 510 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 510 is reduced, thus improving the efficiency of the system.
  • processor 510 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB universal serial bus
  • the charging management module 540 is used for receiving charging input from the charger.
  • the power management module 541 is used for connecting the battery 542 , the charging management module 540 and the processor 510 .
  • the power management module 541 receives the input of the battery 542 and/or the charging management module 540, and supplies power for the processor 510, the internal memory 521, the display screen 594, the camera component 593, and the wireless communication module 560, etc.
  • the wireless communication function of the electronic device can be realized by the antenna 1, the antenna 2, the mobile communication module 550, the wireless communication module 560, the modem processor and the baseband processor.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in an electronic device can be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna may be used in conjunction with a tuning switch.
  • the mobile communication module 550 can provide wireless communication solutions including 2G/3G/4G/5G applied to electronic devices.
  • the mobile communication module 550 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like.
  • the mobile communication module 550 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation.
  • the mobile communication module 550 can also amplify the signal modulated by the modem processor, convert it into electromagnetic wave and radiate it through the antenna 1 .
  • at least part of the functional modules of the mobile communication module 550 may be set in the processor 510 .
  • at least part of the functional modules of the mobile communication module 550 and at least part of the modules of the processor 510 may be set in the same device.
  • the electronic device can communicate with the cloud device through the mobile communication module 550, for example, sending audio and video streams and the like.
  • a modem processor may include a modulator and a demodulator.
  • the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the low-frequency baseband signal is passed to the application processor after being processed by the baseband processor.
  • the application processor outputs a sound signal through an audio device (not limited to a speaker 570A, a receiver 570B, etc.), or displays an image or video through a display screen 594 .
  • the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent of the processor 510, and be set in the same device as the mobile communication module 550 or other functional modules.
  • the electronic device may play audio corresponding to multiple windows through an audio device, and play corresponding video through multiple windows on the display screen 594 .
  • the wireless communication module 560 can provide wireless local area networks (wireless local area networks, WLAN) (such as WiFi network), Bluetooth BT, global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • the wireless communication module 560 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 560 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 510 .
  • the wireless communication module 560 can also receive the signal to be transmitted from the processor 510 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
  • the antenna 1 of the electronic device is coupled to the mobile communication module 550, and the antenna 2 is coupled to the wireless communication module 560, so that the electronic device can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc.
  • GSM global system for mobile communications
  • general packet radio service general packet radio service
  • CDMA code division multiple access
  • WCDMA broadband Code division multiple access
  • time division code division multiple access time-division code division multiple access
  • TD-SCDMA time-division code division multiple access
  • LTE long term evolution
  • BT GNSS
  • the GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • Beidou navigation satellite system beidou navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the electronic device realizes the display function through the GPU, the display screen 594, and the application processor.
  • the GPU is a microprocessor for image processing, connected to the display screen 594 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 510 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the display screen 594 is used to display images, videos and the like.
  • Display 594 includes a display panel.
  • the electronic device may include 1 or N display screens 594, where N is a positive integer greater than 1.
  • the electronic device may render videos in multiple windows through the GPU, and the display screen 594 is used to play corresponding videos through the multiple windows.
  • the electronic device can realize the shooting function through ISP, camera component 593 , video codec, GPU, display screen 594 and application processor.
  • the external memory interface 520 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device.
  • the external memory card communicates with the processor 510 through the external memory interface 520 to implement a data storage function. Such as saving music, video and other files in the external memory card.
  • the internal memory 521 may be used to store computer-executable program codes including instructions.
  • the internal memory 521 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data (such as audio data, phone book, etc.) created during the use of the electronic device.
  • the internal memory 521 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the processor 510 executes various functional applications and data processing of the electronic device by executing instructions stored in the internal memory 521 and/or instructions stored in a memory provided in the processor.
  • the electronic device can realize the audio function through the audio module 570, the speaker 570A, the receiver 570B, the microphone 570C, and the application processor. Such as music playback, recording, etc.
  • the audio module 570, the loudspeaker 570A, the receiver 570B and the microphone 570C, as well as the specific working principles and functions of the buttons 590, the motor 591, the indicator 592 and the SIM card interface 595 you can refer to the introduction in the conventional technology .
  • the software of the electronic device can be divided into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. As shown in Figure 5B, the software structure of the electronic device can be divided into three layers from top to bottom:
  • Application layer (referred to as application layer), application framework layer (referred to as framework layer), system library, Android runtime and kernel layer (also referred to as driver layer).
  • the application layer may include a series of application packages, such as camera, gallery, calendar, call, map, navigation, bluetooth, music, video, short message and other applications.
  • application for short below.
  • the application layer may also include an RTC software development kit (software development kit, SDK) and a call SDK.
  • the call application is mainly used to complete a user interface (user interface, UI) and interaction logic.
  • the call SDK is mainly used to connect with the communication cloud, provide account management, contact management, signaling communication and other capabilities, complete audio and video data collection, broadcasting, encoding and decoding, connect with RTC SDK, and complete the intercommunication of media streams during the call.
  • the RTC SDK is mainly used to interact with the RTC cloud to provide the ability to send media streams (such as audio and video streams).
  • the application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
  • the application framework layer may include a window management server (window manager service, WMS), an activity management server (activity manager service, AMS) and an input event management server (input manager service, IMS).
  • the application framework layer may also include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. (not shown in FIG. 5B ).
  • the system library and the Android runtime include the functions that the FWK needs to call, the Android core library, and the Android virtual machine.
  • a system library can include multiple function modules. For example: browser kernel, three-dimensional (3 dimensional, 3D) graphics, font library, etc.
  • a system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer can include display drivers, input/output device drivers (for example, keyboards, touch screens, earphones, speakers, microphones, etc.), device nodes, camera drivers, audio drivers, and sensor drivers.
  • input/output device drivers for example, keyboards, touch screens, earphones, speakers, microphones, etc.
  • device nodes camera drivers, audio drivers, and sensor drivers.
  • the user performs an input operation through the input device, and the kernel layer can generate corresponding original input events according to the input operation, and store them in the device node.
  • Input/output device drivers can detect user input events. For example, an operation where the user sets the window priority by dragging the window.
  • the end-side device (such as the receiving end) also includes a weak network decision module and a coder/decoder.
  • the weak network decision module is used to predict the overall bandwidth according to the predicted bandwidth, determine whether it is in a weak network environment according to the overall bandwidth and the bandwidth requirements of multiple audio and video streams, and perform weak network decision-making and processing when it is determined to be in a weak network environment.
  • the encoding/decoding module is responsible for decoding audio and video streams.
  • the weak network decision-making module and the encoding/decoding module are located at the application framework layer as an example.
  • the weak network decision module and the codec/decode module can be located at any software architecture layer of the device on the end (such as the receiving end).
  • the above-mentioned weak network decision-making module and encoding/decoding module may also be located at the software architecture layer such as the application program layer, the kernel layer, or the system library of the end-side device (such as the receiving end).
  • a multi-window video communication method provided in the embodiment of the present application may be implemented based on a real time communication (real time communication, RTC) call service architecture.
  • RTC real time communication
  • FIG. 6 shows a schematic diagram of an RTC-based call service architecture.
  • the end-side devices communicate through cloud devices.
  • the architecture includes two end-side devices as an example, but this embodiment of the present application does not limit the specific number of end-side devices in multi-window video communication.
  • cloud devices include communication cloud and RTC cloud.
  • the communication cloud includes an account server and a signaling server.
  • RTC cloud includes RTC server and RTC SFU.
  • the account server is mainly used for storing and maintaining account information, contact information, and push (Push) information.
  • the signaling server is mainly responsible for forwarding call information and call control signaling.
  • the RTC server is mainly responsible for room (Room) access authentication, SFU resource allocation, routing policy, and Room operation/interaction.
  • the RTC SFU is mainly used to maintain the publish/subscribe relationship of media streams, forward media streams (such as audio and video streams), and network adaptability.
  • the end-side device is a sending end (such as a first device) or a receiving end (such as a second device).
  • the call application, call SDK and RTC SDK are installed in the end-side device.
  • the call application is mainly used to complete UI and interaction logic.
  • the call SDK is mainly used to connect with the communication cloud, provide account management, contact management, signaling communication and other capabilities, complete audio and video data collection, broadcasting, encoding and decoding, connect with RTC SDK, and complete the intercommunication of media streams during the call.
  • the RTC SDK is mainly used to interact with the RTC cloud to provide the ability to send media streams (such as audio and video streams).
  • the end-side device also includes an encoding/decoding module.
  • the encoding module is used to encode audio and video streams; when the end-side device is a receiving end, the decoding module is used to decode audio and video streams.
  • the structure of the RTC SFU in the call service architecture shown in FIG. 6 may be the structure of the SFU shown in FIG. 2 .
  • the RTC SFU may include a forwarding module and a bandwidth prediction module.
  • the forwarding module is used to forward audio and video streams.
  • the bandwidth prediction module is used for bandwidth prediction.
  • the terminal-side device (such as the receiving terminal) further includes a weak network decision module.
  • the weak network decision module is used to predict the overall bandwidth (that is, predict bandwidth resources), determine whether it is in a weak network environment according to the overall bandwidth and the bandwidth requirements of multiple audio and video streams, and perform weak network decision-making and processing when it is determined to be in a weak network environment.
  • the weak network decision-making module you can refer to the specific introduction below.
  • FIG. 7 shows a multi-window video communication architecture diagram provided by an embodiment of the present application.
  • RTC SFU 1 and RTC SFU 2 are deployed in a distributed manner to improve network scalability.
  • Each sender shown in Figure 7 can select the optimal RTC SFU for audio and video stream forwarding by itself, for example, select the optimal RTC SFU according to network conditions (such as operators, regions, etc.).
  • sender A chooses to send audio and video stream 1 to receiver D through RTC SFU 1; sender B and sender C choose to send audio and video stream 2 and audio and video to receiver D through RTC SFU 2 respectively Stream 3.
  • the bandwidth prediction module of RTC SFU 1 shown in Figure 7 performs bandwidth prediction according to the downlink data (audio and video stream 1 shown in Figure 7), and sends the bandwidth prediction result to the weak network decision module of the receiving end D.
  • the bandwidth prediction module calculates the bandwidth prediction value 1 according to the data such as time delay and packet loss of the audio and video stream 1 shown in FIG. 7 .
  • the bandwidth prediction module of the RTC SFU 2 shown in Figure 7 calculates the bandwidth prediction value 2 according to the data such as the delay and packet loss of the audio and video stream 2 shown in Figure 7, and sends it according to the audio and video stream 3 shown in Figure 7.
  • the bandwidth prediction value 3 is obtained by calculating the time delay, packet loss and other data.
  • the receiving end D shown in FIG. 7 includes a decoding module and a weak network decision module.
  • the decoding module of the receiving end D is used to decode the received audio and video stream (as shown in FIG. 7 , audio and video stream 1 & audio and video stream 2 & audio and video stream 3).
  • the decoding module may include multiple decoders, and the multiple decoders are used to share the decoding work of audio and video streams.
  • the decoding module includes a decoder A, a decoder B and a decoder C
  • the decoder A is used to decode the audio and video stream 1
  • the decoder B is used to decode the audio and video stream 2
  • the decoder C is used to decode the audio and video stream 3 .
  • the weak network decision-making module shown in Figure 7 is used to carry out overall bandwidth prediction (i.e. bandwidth resource Prediction) to get the total bandwidth prediction value. Further, the weak network decision-making module shown in Figure 7 is also used to make weak network judgments based on the total bandwidth prediction value and the bandwidth requirements of audio and video streams, and when it is determined to be in a weak network environment, the video in the window with a lower priority is preferentially downgraded and subscribed , to ensure the smooth playback of videos in windows with higher priority.
  • bandwidth resource Prediction bandwidth resource Prediction
  • the weak network decision-making module may downgrade the subscription to the video in the lower priority window, which may include but not limited to one or more of the following: unsubscribe, resume subscription, delay subscription, reduce definition, improve clarity degree etc.
  • the third device may also be an electronic device such as a smart phone.
  • the third device can form a peer-to-peer (P2P) network architecture with the first device and the second device, wherein the first device, the second device and the third device can all serve as the receiver.
  • P2P peer-to-peer
  • the third device may also serve as a forwarding device, forwarding the audio and video stream from the first device to the second device.
  • the following will take the multi-window video communication architecture shown in Figure 7 as an example, that is, multiple sending ends (that is, the first device) send audio and video streams to the receiving end (that is, the second device) through the third device (such as SFU) as an example , taking the display interface of the receiving end including the large window, small window 1, and small window 2 as shown in FIG. 4 as an example, a multi-window video communication method provided by the embodiment of the present application is specifically introduced with reference to the accompanying drawings.
  • the multi-window video communication method provided by the embodiment of the present application may include the following steps S801-S804:
  • Multiple sending ends that is, the first device send audio and video streams to a third device (such as the SFU).
  • the audio and video stream also carries the identification (identification, ID) of the receiving end (such as the second device), which is used for the third device (such as SFU) to forward the audio to the receiving end according to the ID of the receiving end. video stream.
  • ID identification
  • the ID of the receiver carried in the audio and video stream is also used by the SFU to predict the downlink bandwidth of the corresponding downlink path.
  • the audio and video stream also carries the ID of the sending end.
  • the above step S801 may specifically include: the sending end A sends the audio and video stream 1 to the RTC SFU 1, and the sending end B and the sending end C send the audio and video stream 1 to the RTC SFU 1 respectively.
  • 2 Send audio and video stream 2 and audio and video stream 3.
  • the audio and video stream 1, the audio and video stream 2 and the audio and video stream 3 carry the ID of the sending end D.
  • the audio and video stream 1 carries the ID of the sender A
  • the audio and video stream 2 carries the ID of the sender B
  • the audio and video stream 3 carries the ID of the sender C.
  • the audio and video streams in this embodiment of the present application may also carry the following information: stream resolution information, frame rate information, encoding Codec, stream level, and the like.
  • audio-video stream 1, audio-video stream 2, and audio-video stream 3 shown in FIG. 9 not only include audio-video information, but also carry information as shown in Table 4 below:
  • the embodiment of the present application does not limit the specific order and timing of sending audio and video streams by multiple sending ends.
  • the sending end A, sending end B, and sending end C shown in FIG. 7 may send audio and video streams at the same time, or may send audio and video streams in any order of time.
  • the third device (such as the SFU) forwards audio and video streams to the receiving end (such as the second device), predicts downlink bandwidth, and sends the bandwidth prediction result to the receiving end.
  • the above step S802 may specifically include: the RTC SFU 1 transfers the voice video stream 1 to the receiving end D, and simultaneously performs bandwidth prediction to obtain a bandwidth prediction value 1, and Send the bandwidth prediction value 1 to the receiving end D; RTC SFU 2 transfers the voice video stream 2 to the receiving end D, and at the same time performs bandwidth prediction to obtain the bandwidth prediction value 2, and sends the bandwidth prediction value 2 to the receiving end D; RTC SFU 2 to the receiving end D forwards the audio and video stream 3, performs bandwidth prediction at the same time to obtain a bandwidth prediction value 3, and sends the bandwidth prediction value 3 to the receiving end D.
  • the predicted bandwidth value 1, the predicted bandwidth value 2, and the predicted bandwidth value 3 shown in FIG. 9 are all 500 kbps.
  • the third device may forward the audio-video stream to the receiver according to the ID of the receiver carried in the audio-video stream.
  • RTC SFU 1 shown in Figure 9 can forward audio and video stream 1 to receiver D according to the ID of receiver D carried in audio and video stream 1 from transmitter A;
  • RTC SFU 2 shown in Figure 9 can transmit audio and video stream 1 based on The ID of the receiving end D carried in the audio and video stream 2 of the terminal B forwards the audio video stream 2 to the receiving end D; and the RTC SFU 2 transmits the ID of the receiving end D to the The receiving end D forwards the audio and video stream 3.
  • the third device (such as the SFU) can predict the downlink bandwidth of the corresponding downlink path according to the ID of the receiver carried in the audio and video stream.
  • the audio and video corresponding to the audio and video streams forwarded by multiple sending ends to the receiving end through a third device are respectively played in multiple windows displayed on the display screen of the receiving end.
  • FIG. 10 shows several examples of multi-window display.
  • (a) in FIG. 10 and (b) in FIG. 10 show the multi-window display interface of the receiving end in a multi-window video communication scenario (such as a multi-party conference scenario or a group video chat scenario).
  • (c) in FIG. 10 shows the multi-window display interface of the receiving end in the online education scenario.
  • FIG. 10 only shows windows related to this application.
  • function keys, menu bars, navigation keys/bars, etc. may also be displayed on the interface of the receiving end, which is not limited by this application.
  • the sending end A, the sending end B, and the sending end C shown in Figure 7 respectively report to the receiving end D
  • the sent audio-video stream 1, audio-video stream 2, and audio-video stream 3 are used to play in the large window, small window 1, and small window 2 of the receiving end D respectively.
  • the SFU may also send the bandwidth requirement to the receiving end.
  • the required bandwidths of audio-video stream 1, audio-video stream 2, and audio-video stream 3 shown in FIG. 9 are 600 kbps, 500 kbps, and 500 kbps respectively.
  • the third device may also send the frame extraction status and the corresponding bandwidth requirement to the receiving end.
  • the frame extraction state is used to indicate whether the video coding mode is to extract frames or not to extract frames.
  • the frame extraction state information and corresponding bandwidth requirements of audio-video stream 1, audio-video stream 2, and audio-video stream 3 shown in FIG. 9 are shown in Table 5 below:
  • Audio and video streaming Frame state bandwidth requirements Audio and video stream 1 no frames 400kbps Audio and video stream 2 Frame 300kbps Audio and video stream 3 Frame 300kbps
  • the receiving end obtains a total bandwidth prediction value according to bandwidth prediction results from one or more third devices (such as SFU).
  • third devices such as SFU
  • the size of the total bandwidth prediction value can be used to represent the amount of bandwidth resources. For example, a larger total bandwidth prediction value indicates more sufficient bandwidth resources; a smaller total bandwidth prediction value indicates less bandwidth resources.
  • the above step S803 may specifically include: the receiving end D according to the bandwidth prediction value 1 from the RTC SFU 1, the bandwidth prediction value 2 from the RTC SFU 2 and The bandwidth prediction value is 3, and the total bandwidth prediction value is obtained.
  • total bandwidth prediction value bandwidth prediction value 1+bandwidth prediction value 2+bandwidth prediction value 3.
  • the predicted bandwidth value 1, the predicted bandwidth value 2, and the predicted bandwidth value 3 shown in FIG. 9 are all 500 kbps, it can be obtained that the total predicted bandwidth value is 1500 kbps.
  • this application does not limit the specific algorithm for the receiving end to obtain the total bandwidth prediction value based on the bandwidth prediction results from one or more SFUs.
  • this part reference can be made to calculation methods in conventional technologies.
  • the receiving end adjusts a subscription strategy for audio and video streams in one or more windows according to priorities corresponding to the multiple windows.
  • the above step S804 may specifically include: when determining a weak network according to the total bandwidth prediction value, the receiving end D according to the large window, small window 1 and small window 2, adjust the subscription strategy for audio and video streams in one or more windows of the large window, small window 1, and small window 2.
  • multiple windows on the display screen of the receiving end may have priority attributes.
  • the priority corresponding to the window is used to represent the importance of the video played in the window to the user at the receiving end. For example, relatively important windows have a higher priority than relatively less important windows.
  • the large window plays high-definition video at a higher resolution (540P), and the small windows 1 and 2 play normal-definition video at a lower resolution (360P).
  • the video played in the large window is more important to the user than the videos played in the small window 1 and the small window 2. Therefore, the priority of the large window shown in FIG. 11 is higher than that of the small window 1 and the small window 2 .
  • the courseware/whiteboard/screen sharing window is the core of online education, while the portrait of the lecturer is relatively unimportant. Therefore, the priority of the courseware/whiteboard/screen sharing window is higher than that of the lecturer portrait window. The specific method for determining the priority will be introduced below.
  • weak network means that under the current network environment, bandwidth resources are insufficient to meet the bandwidth requirements of audio and video streams.
  • the weak network means that under the current network environment, the total bandwidth prediction value obtained by the receiving end (for example, 300kbps-1000kbps) is smaller than the bandwidth value required by the audio and video stream (for example, 1200kbps).
  • the weak network means that under the current network environment, the bandwidth resources are insufficient to meet the video stream frame extraction. previous bandwidth requirements.
  • the video stream corresponding to the large window displayed on the receiving end D requires a bandwidth of 700 kbps before frame extraction
  • the video stream corresponding to small window 1 and window 2 requires a bandwidth of 500 kbps before frame extraction.
  • a weak network means that under the current network environment, bandwidth resources are insufficient to meet the bandwidth requirement of the video stream after frame extraction.
  • the video stream corresponding to the large window displayed on the receiving end D requires a bandwidth of 400 kbps after frame extraction
  • the video stream corresponding to small window 1 and small window 2 requires a bandwidth of 300 kbps before frame extraction.
  • the receiving end can determine that it is in a weak network environment.
  • the embodiment of the present application uses layered coding as an example, and for other video coding methods, the multi-window video communication method provided in the embodiment of the present application is also applicable. And, the embodiment of the present application takes the layered encoding of IDR frame+P frame+B frame, or IDR frame+P frame as an example. For other layered encoding types, the multi-window video communication method provided by the embodiment of the present application is also applicable .
  • the adjustment of the subscription policy by the receiving end may include, but not limited to, one or more of the following: unsubscribing, resuming subscription, delaying subscription, reducing clarity, improving clarity, and the like.
  • the receiving end when the receiving end adjusts the subscription policy, it can only adjust the subscription policy for the video stream, and keep the audio stream playing normally, so as to ensure the normal voice communication and communication of the user and ensure the user experience.
  • unsubscribing refers to unsubscribing from the corresponding video stream to cancel the video display in the window.
  • Restoring the subscription refers to resuming the subscription to the corresponding video stream to restore the video display in the window.
  • Delayed subscription refers to delayed subscription to the corresponding video stream, and the video display in the delayed window.
  • Reducing the definition refers to subscribing to the reduced-definition video stream for the window, for example, switching from subscribing to a high-definition video stream to subscribing to a normal-definition video stream.
  • Raising the definition refers to subscribing to the video stream with improved definition for the window, for example, switching from subscribing to a normal-definition video stream to subscribing to a high-definition video stream.
  • the display interface of the receiving end includes m windows (m ⁇ 3, m is an integer), and all m windows display high-definition video as an example.
  • the specific examples illustrate that the receiving end cancels subscription, resumes subscription, The process of delaying subscription, reducing clarity, or increasing clarity. It should be noted that the embodiment of the present application does not limit the number of windows on the display interface of the receiving end, for example, the display interface of the receiving end may further include 2 windows.
  • the display interface of the receiving end includes windows W 1 , W 2 ... W m (where m represents the priority corresponding to the window), among the windows W 1 , W 2 ... W m , the priority corresponding to W 1 is the highest , the priority corresponding to W 2 is the second, and the priority corresponding to W m is the lowest. If the bandwidth resources are not enough to meet the bandwidth requirements of keeping windows W 1 , W 2 ... W m at the current resolution, but meeting the requirements of keeping window W 1 , W 2 ...
  • the receiving end decides to subscribe to the lowered resolution video stream for the window W m , so as to ensure that the high-priority window (such as The resolution of the video in windows W 1 , W 2 . . . W m-1 ).
  • the receiving end decides to subscribe to the audio and video stream of the first definition for the window, and reduce the subscription to the audio and video stream of the second definition for the window.
  • the second definition is smaller than the first definition.
  • the receiving end decides to reduce the definition of the video in the window W m to ensure the video in the window W 1 , W 2 ??W m-1 HD display. That is, the receiving end decides to subscribe to the normal-definition video stream (that is, the second-definition video stream) for the window W m .
  • RW 1 , RW 2 . . . RW m are bandwidth requirements of windows W 1 , W 2 . . . W m respectively.
  • the resolution of high-definition video may be 540P
  • the resolution of normal-definition video may be 360P.
  • the receiving end after the receiving end reduces the definition of the video in the window W m , if RW 1/HD +RW 2/HD +...+RW m-1/HD +RW m/plain > total bandwidth prediction value ⁇ RW 1/HD +RW 2/HD +...+RW m-1/Pudio +RW m/Pu , then the receiving end can further decide to subscribe to PQ video stream for window W m-1 (that is, the second high-definition video stream), thereby reducing the definition of the video in the window W m-1 to ensure the high-definition display of the video in the windows W 1 , W 2 . . . W m-2 , and so on.
  • window W m-1 that is, the second high-definition video stream
  • the receiving end after the receiving end reduces the video definition in the window W m , if RW 1/HD +RW 2/ HD +...+RW m-1/HD +RW m/Pu >total bandwidth Predicted value ⁇ RW 1/HD +RW 2/HD +...+RW m-1/Pu Qing +RW m/Pu Qing , the receiving end can further decide to cancel the subscription of video stream for window W m , thereby canceling window W m
  • the video in the window is displayed to ensure the high-definition display of the video in the windows W 1 , W 2 . . . W m-2 .
  • the definition may also include three levels of ultra-clear, high-definition and normal definition.
  • the definition when reducing the definition, it can be reduced sequentially according to the gradient of ultra-definition ⁇ high-definition ⁇ normal definition.
  • the display interface of the receiving end includes windows W 1 , W 2 . If the bandwidth requirements for high-definition display are satisfied, but the bandwidth requirements for keeping windows W 1 , W 2 ... W m-1 displayed with the current parameters and the window W m does not display video are met, the receiving end decides to cancel the window W m with the lowest priority Subscribe to the video stream, thereby canceling the display of the video in the lowest priority window to ensure that other windows are displayed in the first definition. .
  • the receiving end decides to cancel the subscription of the video stream for the window instead of subscribing to the second-definition audio and video stream for the window with the lowest priority (that is, to cancel the subscription of the video stream for the window).
  • the second definition is less than or equal to a preset value.
  • the receiving end decides to cancel the video stream subscription for window W m , thereby canceling the video display in window W m , so as to ensure that the windows W 1 , W 2 ... W m-1 HD display of video.
  • the receiver can further decide to subscribe to the video stream with reduced definition (such as normal definition) for window W m-1 , thereby reducing the definition of the video in window W m-1 , or, the receiving end can further decide to cancel the subscription of the video stream for the window W m-1 to ensure the high-definition display of the video in the windows W 1 , W 2 ... W m-2 , and so on.
  • reduced definition such as normal definition
  • the window for unsubscribing may display the last image frame before unsubscribing.
  • the unsubscribe window may display a mask.
  • FIG. 12 shows an example of displaying a mask in the window after the widget 2 with a lower priority (priority is 2) unsubscribes in a weak network environment.
  • the unsubscribed window may display a mask superimposed on the last frame image before unsubscribed.
  • At least one of the windows W 1 , W 2 . . . Wm must be guaranteed to display video in at least one window. For example, at least ensure that the video in window W1 is displayed at the lowest resolution.
  • this application does not limit the specific adjustment strategy adopted by the receiver device for subscribing to audio and video streams when it is in a weak network environment.
  • the bandwidth resources are insufficient to meet the bandwidth requirements of all windows to keep the video stream displayed in the first definition, but the video display in the window with the lowest priority is canceled and other windows display the required bandwidth in the first definition, then the receiving end It may also be decided to cancel the subscription of the video stream for the window with the lowest priority, thereby canceling the display of the video in the window with the lowest priority, so as to ensure that other windows are displayed with the first definition.
  • the receiving end decides to cancel the video stream subscription for window W m , thereby canceling the video display in window W m , so as to ensure that windows W 1 , W 2 ... W m- HD display of videos in 1 .
  • the receiving end if the receiving end monitors the total bandwidth prediction value (ie, bandwidth resource monitoring), and determines that the latest total bandwidth prediction value satisfies the unsubscribed window, the window with the highest priority resumes subscription and other windows remain currently displayed bandwidth requirement (that is, the second preset condition), the receiving end decides to revert to the unsubscribed window, and the window with the highest priority subscribes to the video stream, so as to restore the display of the corresponding video.
  • the total bandwidth prediction value ie, bandwidth resource monitoring
  • the receiver determines that the latest total bandwidth prediction value meets the unsubscribed window within a preset time (such as 6 seconds) through the total bandwidth prediction value monitoring (i.e., bandwidth resource monitoring), priority If the window with the highest priority resumes subscription and other windows maintain the current display bandwidth requirements, the receiving end decides to revert to the unsubscribed window, and the window with the highest priority subscribes to the video stream to resume the display of the corresponding video.
  • a preset time such as 6 seconds
  • bandwidth prediction value monitoring i.e., bandwidth resource monitoring
  • the receiving end decides to revert to subscribing to the video stream for window W m , so as to restore the video display in window W m . For example, the receiving end decides to resume subscribing to the second-definition video for the window W m .
  • the receiving end passes the total bandwidth prediction Value monitoring, confirm that within 6 consecutive seconds, RW 1/ HD +RW 2/HD +...+RW m-1PuD +RW mPuD >total bandwidth prediction value ⁇ RW 1/HD +RW 2/HD +... ...+RW m-1 normal clear , then the receiving end decides to revert to subscribing to the video stream of window W m-1 to restore the video display in window W m-1 , but the status of window W m is still unsubscribed.
  • Delayed subscription means that when a window with a higher priority than a certain window is in the unsubscribed state, it delays subscribing to the video stream for the window, so as to delay the restoration of the video display in the window.
  • window W m-1 For example, assuming that the current windows W 1 , W 2 ... W m-2 all display high-definition video, and windows W m-1 and W m do not display video, then before window W m-1 resumes subscription, the state of window W m Still unsubscribe.
  • the receiving end after reducing the definition, if the receiving end is monitored by the total bandwidth prediction value (ie, bandwidth resource monitoring), and it is determined that the latest total bandwidth prediction value satisfies multiple windows that have been reduced in definition, the priority is the highest.
  • the window improves the definition display and other windows maintain the current display bandwidth requirements (that is, the first preset condition), then the receiving end decides to improve the definition of the video in the window with the highest priority among multiple windows that have been reduced in definition , to ensure the clarity of the video in the high-priority window.
  • the receiving end after reducing the definition, if the receiving end is monitored by the total bandwidth prediction value (i.e., bandwidth resource monitoring), it is determined that within a preset time (such as 6 seconds), the latest total bandwidth prediction value satisfies multiple Among the windows whose resolution has been reduced, the window with the highest priority is displayed with higher resolution and other windows maintain the bandwidth requirements of the current display, then the receiving end decides to increase the resolution of the windows with the highest priority The clarity of the video to ensure the clarity of the video in the high priority window.
  • the total bandwidth prediction value i.e., bandwidth resource monitoring
  • the receiving end decides to increase the definition of the window W m-1 , for example, switching from normal definition display (that is, display with the second definition) to high definition display (that is, display with the first definition).
  • the receiving end decides to improve the definition of window W m-1 , for example, switch from ordinary clear display (that is, display with the second definition) to high-definition display (that is, is displayed in the first definition), but the window Wm still displays the normal definition video (that is, the video in the second definition).
  • the receiving end can adjust the subscription of audio and video streams in multiple windows according to the priorities corresponding to multiple windows. Strategy.
  • the receiving end may decide to reduce the resolution of the video in the window W m-1 while unsubscribing from the window W m (for example, switching from the first resolution display to the second resolution display.)
  • the receiving end may decide to reduce the definition of the video in the window Wm- 1 while reducing the definition of the video in the window Wm -1 . For example, switching from the first definition display to the second definition display.
  • the receiving end can decide to improve the definition of the video in the window Wm -1 while restoring the display of the window Wm in the second definition (for example, switching from the second definition display to the first definition show.)
  • the receiving end may decide to increase the definition of the video in the window Wm- 1 while improving the definition of the video in the window Wm -1 . For example, switching from the second-definition display to the first-definition display.
  • step S805 the method provided by the embodiment of the present application further includes step S805:
  • the receiving end subscribes to multiple sending ends for audio and video streams according to the latest subscription policy.
  • the above step S801 may specifically include: the receiving end D subscribes to the sending end A, the sending end B, and the sending end C according to the determined latest subscription policy. flow.
  • the receiving end may send the following information to the third device to request the third device to subscribe to the corresponding audio and video stream from the first device: the identification (ID) of the sending end, the priority corresponding to the window, the original subscription parameter, the target subscription parameter, the window Identification (Surface ID).
  • ID the identification of the sending end
  • the priority corresponding to the window the priority corresponding to the window
  • the original subscription parameter the target subscription parameter
  • the window Identification the window Identification
  • the receiving end determines to subscribe to the high-definition video stream for the window W m and switch to subscribing for the ordinary clear video stream for the window W m , wherein the sending end of the audio and video stream corresponding to the window W m is the sending end A, and the window W m
  • the receiving end can send the following information to the third device to request the third device to subscribe to the first device for the audio and video stream of the corresponding parameters: ID of sending end A, 3 (priority), high-definition ( original subscription parameter), common clear (target subscription parameter), and the ID of the window W m .
  • the above-mentioned embodiments shown in FIG. 8 and FIG. 9 of this application only use the third device (such as SFU) to predict the downlink bandwidth and send the bandwidth prediction result to the receiving end as an example, and this application does not limit the specific device responsible for downlink bandwidth prediction. equipment.
  • the receiving end that is, the second device
  • the receiving end may also measure and obtain bandwidth prediction results of multiple links during the process of receiving multiple audio and video streams.
  • the above multiple links correspond to multiple audio and video streams.
  • the above multiple links are respectively used to transmit the above multiple audio and video streams.
  • a prompt message can be displayed on the display screen of the electronic device to remind the user that the current network is relatively weak. Difference.
  • FIG. 14 shows an example of displaying a weak network prompt after widget 2 with a lower priority (priority 2) unsubscribes when in a weak network environment.
  • the priority corresponding to the window can be specified by the user, or can be determined by the electronic device itself.
  • the following will introduce several methods of determining the priority corresponding to the window with reference to the accompanying drawings:
  • the user can customize the priority setting by changing the size of the window. For example, a small window is enlarged to a large window, so as to increase the priority corresponding to the window.
  • the priorities of window A, window B, window C and window D shown in (a) in Figure 15 are all 2, in response to the user's operation 1401 of stretching window A into a large window, as shown
  • the electronic device displays a large window D in the middle of the screen, and the electronic device sets the priority corresponding to window D to 1.
  • the user can change the order of the windows by dragging the windows, so as to customize the priority settings. For example, drag a window to the middle of the screen to increase the corresponding priority of the window.
  • the priorities of window B, window C and window D shown in (a) in Figure 16 are all 2, in response to the user's operation 1501 of dragging window D to the middle of the screen, as shown in Figure 16
  • the electronic device displays window D in the middle of the screen, and the electronic device sets the priority corresponding to window D to 1.
  • the embodiment of the present application does not specify a specific manner and method for the user to specify the priority corresponding to the window.
  • the user can also set the priority of the window in the menu.
  • the priority corresponding to the window is determined by the electronic device according to the volume of the audio corresponding to the multiple windows.
  • the electronic device may determine the priorities corresponding to the windows according to the initial volumes of the audios corresponding to the multiple windows.
  • the initial volume of the audio is used to represent the original volume of the audio stream when the electronic device receives the audio stream.
  • the electronic device may adaptively adjust the priority corresponding to the windows according to the initial volume of the audio corresponding to the multiple windows.
  • the electronic device may determine the priority corresponding to the windows according to the playback volume of the audio corresponding to the multiple windows.
  • the user at the receiving end can individually set the playback volume of the audio corresponding to different windows according to their concerns and points of interest. For example, for the window that the user pays most attention to, the user can increase the playback volume, and for the window that the user does not pay attention to, the user can lower the playback volume.
  • the electronic device may adaptively adjust the priority corresponding to the windows according to the playback volume of the audio corresponding to the multiple windows.
  • the priority corresponding to the window is determined by the electronic device according to the functions of the services in the multiple windows.
  • the interface shown in (c) in Figure 10 includes the courseware/whiteboard/screen sharing window and the window where the lecturer portrait is located, wherein the courseware/whiteboard/screen sharing window It is used to display the courseware/whiteboard or display the shared interface, and the window where the lecturer portrait is located is used to play the lecturer video in real time.
  • the core function of online education lies in the display of classroom content. Whether the lecturer portrait is smooth and clear will not affect the acquisition of classroom content. Therefore, the corresponding priority of the courseware/whiteboard/screen sharing window is higher than that of the lecturer portrait window.
  • the above-mentioned electronic device determines the priority corresponding to the window according to the volume of the audio corresponding to the multiple windows or the function of the business in the multiple windows as only two examples.
  • the embodiment of the present application determines the priority corresponding to the window for the electronic device
  • the specific rules and methods are not limited.
  • the priorities corresponding to the windows may also be determined by the electronic device according to other factors such as attributes of videos in multiple windows.
  • multiple windows may correspond to different priorities under different business scenarios or different user requirements.
  • Electronic devices can downgrade and subscribe to video streams based on specific priorities when the downlink bandwidth is limited or the bandwidth fluctuates greatly, such as reducing video resolution, unsubscribing video streams, or delaying subscribing to video streams, etc., to avoid network congestion and ensure Smoothness and/or clarity of video is a high priority.
  • the electronic device determines that the network situation has improved, it can restore the unsubscribed video (that is, restore the subscription) or restore the definition of the downgraded video (that is, improve the definition), so as to ensure maximum protection in multiple windows.
  • the smoothness and/or clarity of video playback are particularly advantageous.
  • serial numbers of the above-mentioned processes do not mean the order of execution, and the order of execution of the processes should be determined by their functions and internal logic, and should not be implemented in this application.
  • the implementation of the examples constitutes no limitation.
  • the electronic device (such as the first device, the second device or the third device) includes corresponding hardware structures and/or software modules for performing various functions.
  • the present application can be implemented in the form of hardware or a combination of hardware and computer software in combination with the units and algorithm steps of each example described in the embodiments disclosed herein. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.
  • the embodiment of the present application can divide the functional modules of the electronic device (such as the first device, the second device or the third device), for example, each functional module can be divided corresponding to each function, or two or more functions can be integrated in one processing module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. It should be noted that the division of modules in the embodiment of the present application is schematic, and is only a logical function division, and there may be other division methods in actual implementation.
  • FIG. 17 it is a structural block diagram of an electronic device provided in the embodiment of the present application.
  • the electronic device may be a first device, a second device or a third device.
  • the electronic device may include a transceiver unit 1710 , a processing unit 1720 and a storage unit 1730 .
  • the transceiver unit 1710 is configured to support the second device to receive audio and video streams from multiple first devices.
  • the transceiver unit 1710 is configured to support the second device to receive audio and video streams from multiple first devices forwarded by the third device.
  • the transceiver unit 1710 may also be configured to support the second device to receive bandwidth prediction values corresponding to multiple audio and video streams from the third device.
  • the transceiver unit 1710 may also be used to support the second device to subscribe to the audio and video stream from the first device, and/or other processes related to the embodiment of the present application.
  • the processing unit 1720 is configured to support the second device to obtain a total bandwidth prediction value according to multiple bandwidth prediction results, determine that it is in a weak network environment according to the total bandwidth prediction value, and adjust a subscription strategy for audio and video streams in one or more windows. In some embodiments, the processing unit 1720 is further configured to support the second device to measure bandwidth prediction results corresponding to multiple audio and video streams, and/or other processes related to the embodiments of the present application.
  • the storage unit 1730 is used to store computer programs and implement processing data and/or processing results in the methods provided by the embodiments of the present application.
  • the transceiver unit 1710 may include a radio frequency circuit.
  • the electronic device can receive and send wireless signals through a radio frequency circuit.
  • radio frequency circuitry includes, but is not limited to, an antenna, at least one amplifier, transceiver, coupler, low noise amplifier, duplexer, and the like.
  • radio frequency circuits can also communicate with other devices through wireless communication.
  • the wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile Communications, General Packet Radio Service, Code Division Multiple Access, Wideband Code Division Multiple Access, Long Term Evolution, Email, Short Message Service, etc.
  • each module in the electronic device may be implemented in the form of software and/or hardware, which is not specifically limited. In other words, electronic equipment is presented in the form of functional modules.
  • the "module” here may refer to an application-specific integrated circuit ASIC, a circuit, a processor and memory executing one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the above-mentioned functions.
  • the computer program product includes one or more computer instructions.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).
  • the computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated.
  • the available medium can be a magnetic medium, (such as a floppy disk, a hard disk, etc. , tape), optical media (such as digital video disk (digital video disk, DVD)), or semiconductor media (such as solid state disk (SSD)), etc.
  • the steps of the methods or algorithms described in conjunction with the embodiments of the present application may be implemented in hardware, or may be implemented in a manner in which a processor executes software instructions.
  • the software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, mobile hard disk, CD-ROM or any other form of storage known in the art medium.
  • An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium.
  • the storage medium may also be a component of the processor.
  • the processor and storage medium can be located in the ASIC.
  • the ASIC may be located in the electronic device.
  • the processor and the storage medium can also exist in the electronic device as discrete components.

Abstract

La présente demande concerne le domaine technique des communications, et divulgue un procédé, un dispositif et un système de communication vidéo multifenêtre. Dans un processus d'appel vidéo à plusieurs parties, des ressources de bande passante peuvent être pleinement utilisées pour éviter une congestion de réseau, ce qui permet d'assurer la fluidité et/ou la définition d'une vidéo. Selon la présente demande, de multiples fenêtres sont affichées sur une interface d'une extrémité de réception, et lorsque des ressources de bande passante sont insuffisantes pour satisfaire aux exigences de bande passante de multiples flux audio/vidéo (c'est-à-dire, dans un environnement de réseau faible), l'extrémité de réception peut effectuer un abonnement de qualité inférieure selon des priorités particulières correspondant aux multiples fenêtres, telle que la réduction de la définition d'une vidéo, le désabonnement d'un flux vidéo ou le retard de l'abonnement à un flux vidéo, de telle sorte que la congestion du réseau est évitée, et la fluidité et/ou la définition d'une vidéo à haute priorité est assurée.
PCT/CN2022/109423 2021-08-03 2022-08-01 Procédé, dispositif et système de communication vidéo multifenêtre WO2023011408A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110887044.0A CN115706829A (zh) 2021-08-03 2021-08-03 一种多窗口视频通信方法、设备及系统
CN202110887044.0 2021-08-03

Publications (1)

Publication Number Publication Date
WO2023011408A1 true WO2023011408A1 (fr) 2023-02-09

Family

ID=85154361

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/109423 WO2023011408A1 (fr) 2021-08-03 2022-08-01 Procédé, dispositif et système de communication vidéo multifenêtre

Country Status (2)

Country Link
CN (1) CN115706829A (fr)
WO (1) WO2023011408A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117135364A (zh) * 2023-10-26 2023-11-28 深圳市宏辉智通科技有限公司 一种视频解码方法和系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009033348A (ja) * 2007-07-25 2009-02-12 Toshiba Corp ビデオ会議アプリケーションサーバ、ビデオ会議方法およびプログラム
CN101557495A (zh) * 2009-05-18 2009-10-14 上海华平信息技术股份有限公司 视频会议系统的带宽控制方法
US20140317532A1 (en) * 2013-03-15 2014-10-23 Blue Jeans Network User interfaces for presentation of audio/video streams
CN109218653A (zh) * 2018-09-30 2019-01-15 广州视源电子科技股份有限公司 一种视频会议的多窗口显示方法、装置、设备和系统
CN112104880A (zh) * 2020-08-31 2020-12-18 广州华多网络科技有限公司 网络连线直播控制、显示方法及装置、设备、存储介质
US10999344B1 (en) * 2020-06-15 2021-05-04 Google Llc Dynamic video resolution and quality for improved video conferencing
CN113014858A (zh) * 2021-03-05 2021-06-22 深圳壹秘科技有限公司 一种改变分辨率的方法、系统和装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009033348A (ja) * 2007-07-25 2009-02-12 Toshiba Corp ビデオ会議アプリケーションサーバ、ビデオ会議方法およびプログラム
CN101557495A (zh) * 2009-05-18 2009-10-14 上海华平信息技术股份有限公司 视频会议系统的带宽控制方法
US20140317532A1 (en) * 2013-03-15 2014-10-23 Blue Jeans Network User interfaces for presentation of audio/video streams
CN109218653A (zh) * 2018-09-30 2019-01-15 广州视源电子科技股份有限公司 一种视频会议的多窗口显示方法、装置、设备和系统
US10999344B1 (en) * 2020-06-15 2021-05-04 Google Llc Dynamic video resolution and quality for improved video conferencing
CN112104880A (zh) * 2020-08-31 2020-12-18 广州华多网络科技有限公司 网络连线直播控制、显示方法及装置、设备、存储介质
CN113014858A (zh) * 2021-03-05 2021-06-22 深圳壹秘科技有限公司 一种改变分辨率的方法、系统和装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117135364A (zh) * 2023-10-26 2023-11-28 深圳市宏辉智通科技有限公司 一种视频解码方法和系统
CN117135364B (zh) * 2023-10-26 2024-02-02 深圳市宏辉智通科技有限公司 一种视频解码方法和系统

Also Published As

Publication number Publication date
CN115706829A (zh) 2023-02-17

Similar Documents

Publication Publication Date Title
WO2021004381A1 (fr) Procédé d'affichage de vidéocapture d'écran, et appareil électronique
KR101634500B1 (ko) 미디어 작업부하 스케줄러
US8983555B2 (en) Wireless communication techniques
WO2021244341A1 (fr) Procédé et appareil de codage d'image, dispositif électronique et support de stockage lisible par ordinateur
US8947492B2 (en) Combining multiple bit rate and scalable video coding
US20140198838A1 (en) Techniques for managing video streaming
KR20180031547A (ko) 서버에서 멀티 비트 레이트 스트림 미디어를 적응적으로 제공하기 위한 방법 및 장치
US8842159B2 (en) Encoding processing for conferencing systems
CN113556598A (zh) 多窗口投屏方法及电子设备
CN110865782B (zh) 数据传输方法、装置及设备
JP2011176827A (ja) テレビ会議システムの処理方法、テレビ会議システム、プログラム及び記録媒体
WO2023011408A1 (fr) Procédé, dispositif et système de communication vidéo multifenêtre
CN114600468A (zh) 将复合视频流中的视频流与元数据组合
JP2017520940A (ja) 階層符号化されたコンテンツを多重化するための方法および装置
CN116204308A (zh) 音视频算力动态调节方法和装置、电子设备
CN113726815A (zh) 一种动态调整视频的方法和装置
US9467655B2 (en) Computer readable recording medium, communication terminal device and teleconferencing method
WO2021093882A1 (fr) Procédé de réunion vidéo, terminal de réunion, serveur et support de stockage
CN114697731B (zh) 投屏方法、电子设备及存储介质
WO2021223577A1 (fr) Procédé de traitement vidéo, appareil associé, support de stockage, et produit programme
CN117193685A (zh) 投屏数据的处理方法、电子设备及存储介质
CN114257771A (zh) 一种多路音视频的录像回放方法、装置、存储介质和电子设备
US11290680B1 (en) High-fidelity freeze-frame for precision video communication applications
US11936698B2 (en) Systems and methods for adaptive video conferencing
CN117412140A (zh) 用于视频传输的方法及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22852120

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE