WO2022199853A1 - Method for selecting one or more background objects for a video stream - Google Patents

Method for selecting one or more background objects for a video stream Download PDF

Info

Publication number
WO2022199853A1
WO2022199853A1 PCT/EP2021/058002 EP2021058002W WO2022199853A1 WO 2022199853 A1 WO2022199853 A1 WO 2022199853A1 EP 2021058002 W EP2021058002 W EP 2021058002W WO 2022199853 A1 WO2022199853 A1 WO 2022199853A1
Authority
WO
WIPO (PCT)
Prior art keywords
participants
participant
background
background object
information
Prior art date
Application number
PCT/EP2021/058002
Other languages
French (fr)
Inventor
Peter ÖKVIST
Tommy Arngren
Original Assignee
Telefonaktiebolaget Lm Ericsson (Publ)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonaktiebolaget Lm Ericsson (Publ) filed Critical Telefonaktiebolaget Lm Ericsson (Publ)
Priority to PCT/EP2021/058002 priority Critical patent/WO2022199853A1/en
Publication of WO2022199853A1 publication Critical patent/WO2022199853A1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/14Systems for two-way working
    • H04N7/141Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/147Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/14Systems for two-way working
    • H04N7/141Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/142Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
    • H04N2007/145Handheld terminals

Definitions

  • the technology disclosed herein relates generally to the field of radio resource optimization, and in particular to selection of background objects for a video stream in a resource efficient way.
  • Digital meetings in which applications such as, for instance, Zoom, Skype and Teams are used, have become an increasingly common way of having business meetings as well as personal meetings.
  • the increasing number of digital meetings correspondingly puts an increasing load on managing servers and on hosting and conveying infrastructures.
  • the privacy entails a large number of issues, such as when to show sensitive material, what background a participant is able to view behind all other participants, material to share etc.
  • the objective is achieved by a selection method for selecting one or more background objects for a video stream between two or more participants of a digital meeting.
  • the selection method is performed by one or more entities.
  • the method comprises identifying information on at least one background object of a background captured by a camera.
  • the background is to be shared between two or more participants of a set of participants of the digital meeting.
  • the background object is associated with a first participant of the set of participants.
  • the method comprises selecting, based on the identified information, the at least one background object to be provided or to be excluded in the video stream from the first participant and to be shared with at least a second participant of the set of participants.
  • the objective is achieved by a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according to the first aspect.
  • the objective is achieved by computer program product comprising a computer program according to the second aspect and a computer readable means on which the computer program is stored.
  • the computer readable means could be a non-transitory computer readable storage medium.
  • the objective is achieved by a method performed by a managing unit for managing communication resources of a video stream between two or more participants of a digital meeting.
  • the method comprises obtaining information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants.
  • the at least one background object is associated with a first participant of the set of participants.
  • the method further comprises assigning resources to the first participant based on the obtained information.
  • the objective is achieved by a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according the fourth aspect.
  • the objective is achieved by a computer program product comprising a computer program according to the fifth aspect and a computer readable means on which the computer program is stored.
  • the computer readable means could be a non-transitory computer readable storage medium.
  • the objective is achieved by a system configured to select background objects for a video stream between two or more participants of a digital meeting.
  • the system comprises one or more entities configured to identify information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants.
  • the at least one background object is associated with a first participant of the set of participants.
  • the one or more entities are further configured to select, based on the identified information, the at least one background object to be provided in or to be excluded from the video stream from the first participant and shared with at least a second participant of the set of participants.
  • these aspects of the inventive concept maybe used in, for instance, video conferencing, for saving scarce network resources and for improved privacy settings for users of the video conferencing services.
  • Figure l illustrates schematically an environment in which embodiments of the present teachings may be implemented.
  • Figure 2 is a flow chart of embodiments of a method in one or more entities.
  • Figure 3 is a flow chart of embodiments of a method in a managing unit.
  • Fig. 4 is a schematic diagram showing functional units of one or more entities according to an embodiment.
  • Fig. 5 is a schematic diagram showing functional modules of one or more entities according to an embodiment.
  • Fig. 6 is a schematic diagram showing functional units of a managing unit according to an embodiment.
  • Fig. 7 is a schematic diagram showing functional modules of a managing unit according to an embodiment.
  • Fig. 8 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.
  • Web conferencing is an umbrella term for various types of online conferencing and collaborative services such as e.g. web seminars (webinars), webcasts and web meetings.
  • This type of services is known under a variety of different names, such as online meeting, digital meeting, video meeting, e-meeting and virtual meeting.
  • the present teachings are applicable to any such type of service.
  • What specific personal information a first user is comfortable with sharing may, for instance, depend on meeting context, e.g. whether it is private or business, on cultural aspects, or what may or may not be associated with trigger words in respective context.
  • the relation to other meeting attendees may also be a factor to consider. For instance, in private life among trusted friends of known faiths, religion, etc., the first user may be comfortable sharing backdrop ornaments and would even like to put more details for certain objects (e.g. accentuate, emphasize) that are of high personal value, and such action may furthermore even serve to establish common ground among meeting participants, whereas on the other hand, in e.g. business context or in situations with unknow meeting participants, such personal details should be suppressed on perhaps even concealed entirely.
  • the present disclosure provides methods and means for accomplishing this in a resource efficient way, i.e. to enable a user to select what to be shown to whom.
  • the present teachings provide means for a user and for an owner of a managing system to adaptively reduce the amount of consumed resources that are associated with transferring information, or “pixels”, corresponding to objects in the user’s digital meeting background.
  • FIG. l illustrates schematically an environment in which embodiments of the present teachings may be implemented.
  • a system l is shown, which comprises a cloud execution environment 2 wherein a number of network functions (NF) and dedicated functions (DF) reside.
  • NF network functions
  • DF dedicated functions
  • Such network functions may handle a variety of functions, e.g. mobility in a wireless and/ or wired network, security, routing.
  • the network functions may be running on virtualized hardware and not be tied to specific hardware node, whereby they can be executed in different places depending on a scenario at hand and corresponding requirements.
  • one or more entities 7a, 7b may be implemented as such network functions, configured to perform embodiments of methods to be described, e.g. in relation to figure 2.
  • the network infrastructure 3 may be a network for handling wireless communication and various functions thereof.
  • the network infrastructure 3 may then comprise a wired part connected to entities 7a, 7b of the cloud execution environment 2, and a wireless part comprising a wireless communications system 4, in turn comprising e.g. radio nodes 8a, 8b serving a number of wireless user equipment 5a, 5b.
  • the radio nodes 8a, 8b may comprise user equipment 5a, 5b, relay nodes and access nodes.
  • the access nodes are shown separated into antennas, radio units (RU) and digital units (DU).
  • the network infrastructure 3 may support at least one 3GPP cellular communication standard, e.g.
  • the one or more entities 7a, 7b may, in different embodiments, be located either in the cloud execution environment 2 or in the network infrastructure 3, e.g. in a managing server or a controlling node.
  • the entities 7a, 7b when comprising at least two, maybe distributed such that one or more entities is located in the cloud execution environment 2, and the remaining one or more in the network infrastructure 3.
  • ML machine- learning
  • a users’ device 5a, 5b may run a meeting application that is connected to at least one managing unit 6.
  • the managing unit 6 may control inbound/outbound video streams to/from its managed users.
  • the managing unit 6 maybe part of a managing server 9.
  • the managing server 9 may hold capabilities that dispatches radio interface resources among the cellular connected users in terms of e.g. QoS contracts, SLAs, resource allocation (i.e. scheduling of e.g. PRB allocation, etc.); a radio resource managing entity may typically be located in an eNB/gNB and maybe denoted as scheduler.
  • the managing unit 6 may also hold capabilities of object identification and classification in terms of identifying/classifying objects in respective users’ background imagery conveyed from digital meeting applications running in user devices 5a, 5b.
  • the application and/or managing unit 6 may furthermore have capabilities of, for a specific identified and classified object, apart from determining its type, e.g. painting, furniture etc., also determine its money value, its corresponding trigger-value. Such trigger value may for instance comprise grade of controversial, “profanity”, etc.
  • the application and/ or managing unit 6 may furthermore have a capability of substituting the specific object with a generic object of same type. Object values, etc., maybe obtained from external sources, e.g. internet servers, etc., located e.g. in the cloud environment 2.
  • the managing unit 6 may also hold capabilities of determining meeting participants inter-relations and context of the meeting.
  • the managing unit 6 may, for example, identify which of the participants know each other prior to the meeting.
  • the managing unit 6 may perform such identification e.g. through assessment of emails, chats, social medias, friend/VIP-settings, etc.
  • the managing unit 6 may determine if the meeting is in a private or in a business/corporate context. Such determination may, e.g., be performed via email-addresses, based on time of day, and/ or meeting subject, etc.
  • the managing entity 6 may by similar means assess individual participants emails, chats, social medias flows, friend interactions, web browsing patterns, etc. and classify users’ respective personal interest, preferences, etc.
  • the above determinations made by the managing unit 6 may thus be accomplished by the managing unit 6 comprising means for implementing a Machine Learning (ML)- based identification and classification of backdrop/background objects associated with.
  • ML Machine Learning
  • managing unit 6 may deduct, for instance, one or more of:
  • the managing entity 6 may determine: what object that maybe given additional resolution, e.g. in combination with improved frame rate, color depth, etc., where the numbers of objects to accentuate typically is smaller than number of all possible objects in the user’s background image; - which other “complementary objects” in respective background that may be given less accentuating, in terms of e.g. lower frame rate, resolution or color depth, etc.;
  • a video encoding setting that corresponds to requirements to accentuate identified objects and corresponds to necessity to provide a lowest quality (e.g. resolution, etc.) to non-accentuated complementary background objects; in context of video (device) applications, full P-frames to update the image may not be used, but instead a smaller part that is selected dynamically depending on how much of the image that is being subject to changes between different frames.
  • an arbitrary slice size and shape may be used; smallest entity for a macro block in H264 is typically in order or 8x8 (or 16x16) pixels.
  • radio resources are made free, i.e. not allocated for the digital meeting purpose, and may therefore be used by the managing server 9 for other communication purposes.
  • User A Likes watches and has several items on display in the back of the room, and also has a large window facing his garden.
  • the managing unit 6 may hold information about the users A, B, C and in particular about their respective preferences.
  • the managing unit 6 may detect and identify respective objects, in this example trees in the garden and the displayed watches in User A’s backdrop.
  • the managing unit 6 may determine that video rendering effort may be spent on providing User B with better resolution on one set of selected preferred objects in A’s backdrop, e.g. trees and flowers, while User C is provided with better resolution on another set of preferred objects in A’s backdrop, e.g. watches.
  • the managing unit 6 may provide respective receiving users with a video stream from a first user that has improved rendering/resolution on objects determined “liked”, and potentially the opposite on objects found “disliked”.
  • the first user does not have to figure out what in his/her backdrop that may be shown, may be non-disclosed and applied “more rendering to”, the managing entity 6 may more effort/less effort on different objects depending on individual receivers’ preference and sender/individual receivers’ relation.
  • a part of the managing unit 6 may then determine new video encoding reflecting certain accentuated/non-accentuated objects, determine corresponding to the video encoding and QoS/SLA, determine a scheduling policy and associated radio interface resources, where said dedicated per user -user personalized media flow are provided to/from said individual users.
  • Objects subject to accentuation rules may, for instance, comprise book titles, paintings, furniture, etc.; human faces, e.g. family members; items, objects, etc. of interest determined from an ML model of recipient interests, and/or relevant to ongoing discussion of the ongoing meeting (or preceding meeting).
  • Accentuation rules may be applied manually according to first user’s preferences (“I’d like to emphasize my painting”), or they may be based on ML model identifying ⁇ triggerword> topics given profiles of meeting participants’ relation to “baseline”.
  • the managing entity 6 may: apply the selected digital accentuation to a certain selected digital object.
  • Such digital accentuation may comprise the managing server to render identified object with higher image/video quality, higher resolution/rate, etc. - improve objects visibility, readability and/ or objects relative relation to other non-accentuated objects in the media stream,
  • the managing entity 6 may provide objects relevant to media streams receivers' interest and preferences and/or objects relevant to ongoing discussion between communication parties
  • User A likes watches and paintings and have several on display in the room.
  • User B is a trusted friend of User A.
  • User C is not a person trusted by User A but is someone that User A and User B is discussing a deal with about a used car.
  • Their respective devices at least for User A, maybe used for scanning their respective rooms. For instance, perform a 3D scanning of places and objects. This can be performed using e.g. Lidar capabilities (e.g. determining ranges by targeting objects with a laser and measuring return time for reflected light), and creating an USDZ-file or similar, with a dictionary with location/ description of all identified backdrop objects. Further the device maybe used to create a reference room with the same type of objects. Then, for each object create an object-attribute-metric lookup, e.g. describing:
  • value either from a public data base (e.g. watches: chrono24.com), or user- provided in an initial setup procedure, or user-prompted given that no value- object lookup is identified,
  • the managing unit 6 may further determine that certain detected objects may be fully disclosed in User B’s media stream, that certain detected objects should not be disclosed in User C’s media stream.
  • the managing unit 6 may further select disclosure rules according to User A’s no trust in User C, e.g. rules such as:
  • the managing unit 6 may further select disclosure rule according to A’s full trust in B where no digital object overlay at all is applied.
  • a part of the managing server may then determine new video encoding reflecting certain accentuated/non-accentuated objects, determine corresponding to said video encoding and QoS/SLA, determine a scheduling policy and associated radio interface resources, where said dedicated per user-user personalized media flow are provided to/from said individual users. Further details in context of partial object disclosure, full object alike-overlay are described in the following.
  • Objects identified in the media stream maybe subject to disclosure rules:
  • Objects subject to disclosure rules may comprise: Book titles, paintings, furniture, etc., human faces, e.g. family members.
  • the disclosure rules may be applied manually according to the first user’s preferences, and/or applied in accordance with relations between first user and other meeting participants (family, corporate, unknown), and/or based on an ML model identifying ⁇ triggerword> topics given profiles of meeting participants’ relation to “baseline”, and/or an ML model maybe trained with respect to potential meeting participants according to: cultures, religion, associated use & manners, gender, age, etc.
  • the managing entity 6 may:
  • Another aspect of the disclosure is to have a local application in the user device that analyzes background objects and/or sounds.
  • the local application may use known preferences related to each participant in an online meeting to prioritize and/ or remove part of background information that may not be relevant for all participants.
  • the local application may use known preferences related to each participant in an online meeting to prioritize and share same background information to all participants; this step may require that individual local applications are communicating, and share said user preferences.
  • the application may prioritize background information and share part that are relevant for each participant.
  • Figure 2 is a flow chart of embodiments of a method 20 for selecting at least one background object for a video stream between at least two participants of a digital meeting.
  • the method 20 is performed in one or more entities of a system 1 as described in relation to figure 1.
  • a method 20 is provided for selecting one or more background objects for a video stream between two or more participants of a digital meeting.
  • the selection method 20 is performed by one or more entities 7a, 7b.
  • the method 20 comprises identifying 22 information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants.
  • the background object is associated with a first participant of the set of participants.
  • the method 20 further comprises selecting 24, based on the identified information, the at least one background object to be provided or to be excluded in the video stream from the first participant and to be shared with at least a second participant of the set of participants.
  • the identifying 22 comprises using machine learning for identifying and classifying 23 background objects associated with at least one of the two or more participants.
  • the identifying 22 comprises identifying the at least one background object based on one or more of: type of object, economic value of object and grade of controversiality of object.
  • the identifying 22 comprises identifying the at least one background object based on one or more of: preferences of the first participant, preferences of the at least second participant, central analysis of controversiality of the object and central analysis of economic value of the object.
  • the selecting 24 comprises selecting the at least one background object based on one or more of: relation of the first participant to the at least second participant, meeting context, degree of interest for the one or more background objects and cross-intersection between three or more of the set of participants.
  • the method 20 comprises providing 26 the information on the one or more background objects to a managing unit 6 for managing use of resources of the conveying of the video stream.
  • FIG. 3 is a flow chart of embodiments of a method in managing unit 6.
  • the method 30 is performed in a managing unit 6 for managing communication resources of a video stream between two or more participants of a digital meeting.
  • the method 30 comprises obtaining 32 information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the at least one background object being associated with a first participant of the set of participants.
  • the method 30 further comprises assigning 34 resources to the first participant based on the obtained information.
  • the obtaining 32 comprises receiving the information on the one or more background objects from one or more entities 7a, 7b of a cloud execution environment 2.
  • the obtained information comprises determined and classified background object preferences for at least the first and the second participants of the set of participants.
  • the method 30 then comprises determining, based on the background object preferences, one or more of: resolution, frame rate and color depth, based on degree of accentuation determined for a respective participant of the set of participants.
  • the assigning 34 comprises assigning to the first participant amount of radio interface resources for one or more video streams based on one or more of: determined accentuating or non-accentuating video encoding, and determined per-user radio resource demand in accordance to video-encoding.
  • the resources comprise radio resources required for at least part of a path of the video stream.
  • the obtained information comprises one or more of: relation of the first participant to the at least second participant, meeting context, degree of interest of the at least second participant for the at least one background object and cross-intersection between three or more of the set of participants.
  • the managing unit 6 comprises a node of a network infrastructure 3 or a node 8a, 8b of an access network of a wireless communications network.
  • Fig. 4 schematically illustrates, in terms of a number of functional units, the components of one entity of the one or more entities according to an embodiment.
  • Processing circuitry 110 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 500a (as in Fig. 8), e.g. in the form of a storage medium 520.
  • the processing circuitry no may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the processing circuitry no is configured to cause the entity to perform a set of operations, or steps, as disclosed above.
  • the storage medium 130 may store the set of operations, and the processing circuitry no may be configured to retrieve the set of operations from the storage medium 130 to cause the entity to perform the set of operations.
  • the set of operations maybe provided as a set of executable instructions.
  • the processing circuitry no is thereby arranged to execute methods as herein disclosed.
  • the storage medium 230 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.
  • the entity may further comprise a communications interface 120 at least configured for communications with other entities, functions, nodes, and devices.
  • the communications interface 120 may comprise one or more transmitters and receivers, comprising analogue and digital components.
  • the processing circuitry 110 controls the general operation of the entity e.g. by sending data and control signals to the communications interface 120 and the storage medium 130, by receiving data and reports from the communications interface 120, and by retrieving data and instructions from the storage medium 130.
  • Other components, as well as the related functionality, of the entity are omitted in order not to obscure the concepts presented herein.
  • Fig. 5 is a schematic diagram showing functional modules of one or more entities according to an embodiment.
  • Fig. 5 schematically illustrates, in terms of a number of functional modules, the components of an entity according to an embodiment.
  • the entity of Fig. 5 comprises a number of functional modules; an identify module 210 configured to perform step 22, and a select module 220 configured to perform step 24.
  • the entity of Fig. 5 may further comprise a number of optional functional modules, such as a provide module 230 configured to perform step 26.
  • each functional module 210, 220, 230 may be implemented in hardware or in software.
  • one or more or all functional modules 210, 220, 230 maybe implemented by the processing circuitry no, possibly in cooperation with the communications interface 120 and the storage medium 130.
  • the processing circuitry 110 may thus be arranged to from the storage medium 130 fetch instructions as provided by a functional module 210, 220, 230 and to execute these instructions, thereby performing any steps of the entity as disclosed herein.
  • Fig. 6 is a schematic diagram showing functional units of a managing unit 6 according to an embodiment.
  • Processing circuitry 310 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 500b (as in Fig. 8), e.g. in the form of a storage medium 520.
  • the processing circuitry 510 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the processing circuitry 310 is configured to cause the managing unit 6 to perform a set of operations, or steps, as disclosed above.
  • the storage medium 330 may store the set of operations, and the processing circuitry 310 may be configured to retrieve the set of operations from the storage medium 330 to cause the managing unit 6 to perform the set of operations.
  • the set of operations maybe provided as a set of executable instructions.
  • the processing circuitry 310 is thereby arranged to execute methods as herein disclosed.
  • the storage medium 330 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.
  • the managing unit 6 may further comprise a communications interface 320 at least configured for communications with other entities, functions, nodes, and devices.
  • the communications interface 320 may comprise one or more transmitters and receivers, comprising analogue and digital components.
  • the processing circuitry 310 controls the general operation of the entity e.g. by sending data and control signals to the communications interface 320 and the storage medium 330, by receiving data and reports from the communications interface 320, and by retrieving data and instructions from the storage medium 330.
  • Other components, as well as the related functionality, of the entity are omitted in order not to obscure the concepts presented herein.
  • Fig. 7 schematically illustrates, in terms of a number of functional modules, the components of a managing unit according to an embodiment.
  • each functional module 410, 420 may be implemented in hardware or in software.
  • one or more or all functional modules 410, 420 may be implemented by the processing circuitry 310, possibly in cooperation with the communications interface 320 and the storage medium 330.
  • the processing circuitry 310 may thus be arranged to from the storage medium 330 fetch instructions as provided by a functional module 410, 420 and to execute these instructions, thereby performing any steps of the managing unit as disclosed herein.
  • Fig. 8 shows one example of a computer program product 500a, 500b comprising computer readable means 520.
  • a computer program 510a can be stored, which computer program 510a can cause the processing circuitry 110 and thereto operatively coupled entities and devices, such as the communications interface 120 and the storage medium 130, to execute methods according to embodiments described herein.
  • the computer program 510a and/or computer program product 500a may thus provide means for performing any steps of the one or more entities as herein disclosed.
  • a computer program 510b can be stored, which computer program 510b can cause the processing circuitry 310 and thereto operatively coupled entities and devices, such as the communications interface 320 and the storage medium 330, to execute methods according to embodiments described herein.
  • the computer program 510b and/ or computer program product 500b may thus provide means for performing any steps of the managing unit as herein disclosed.
  • the computer program product 500a, 500b is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu- Ray disc.
  • the computer program product 500a, 500b could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • the computer program 510a, 510b is here schematically shown as a track on

Abstract

A selection method is disclosed for selecting one or more background objects for a video stream between two or more participants of a digital meeting. The selection method is performed by one or more entities, and comprises: identifying (22) information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the background object being associated with a first participant of the set of participants, and selecting (24), based on the identified information, the at least one background object to be provided or to be excluded in the video stream from the first participant and to be shared with at least a second participant of the set of participants.

Description

Method for selecting one or more background objects for a video stream Technical field
The technology disclosed herein relates generally to the field of radio resource optimization, and in particular to selection of background objects for a video stream in a resource efficient way.
Background
Digital meetings, in which applications such as, for instance, Zoom, Skype and Teams are used, have become an increasingly common way of having business meetings as well as personal meetings. The increasing number of digital meetings correspondingly puts an increasing load on managing servers and on hosting and conveying infrastructures.
Such meetings are often made, at least partly over wireless communication links and oftentimes entails resource demanding features, such as e.g. video. It follows that there is an increasing need for wireless network operators to use their resources as efficiently as possible in order to ensure that there is no lack of capacity in the communication system nor that latency increases, as both result in decreased user satisfaction.
Another aspect of the digital meetings is the privacy of the users. The privacy entails a large number of issues, such as when to show sensitive material, what background a participant is able to view behind all other participants, material to share etc.
There is thus a need for meeting various requirements for digital meetings to be held in a satisfying manner, e.g. in view of ensuring sufficiency of resources, avoiding latency in the infrastructure and privacy.
Summary An objective of embodiments disclosed herein is to solve or at least alleviate at least one of the above-mentioned problems.
According to a first aspect the objective is achieved by a selection method for selecting one or more background objects for a video stream between two or more participants of a digital meeting. The selection method is performed by one or more entities. The method comprises identifying information on at least one background object of a background captured by a camera. The background is to be shared between two or more participants of a set of participants of the digital meeting. The background object is associated with a first participant of the set of participants. The method comprises selecting, based on the identified information, the at least one background object to be provided or to be excluded in the video stream from the first participant and to be shared with at least a second participant of the set of participants.
According to a second aspect the objective is achieved by a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according to the first aspect.
According to a third aspect the objective is achieved by computer program product comprising a computer program according to the second aspect and a computer readable means on which the computer program is stored. The computer readable means could be a non-transitory computer readable storage medium.
According to a fourth aspect the objective is achieved by a method performed by a managing unit for managing communication resources of a video stream between two or more participants of a digital meeting. The method comprises obtaining information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants. The at least one background object is associated with a first participant of the set of participants. The method further comprises assigning resources to the first participant based on the obtained information.
According to a fifth aspect the objective is achieved by a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according the fourth aspect.
According to a sixth aspect the objective is achieved by a computer program product comprising a computer program according to the fifth aspect and a computer readable means on which the computer program is stored. The computer readable means could be a non-transitory computer readable storage medium. According to a seventh aspect the objective is achieved by a system configured to select background objects for a video stream between two or more participants of a digital meeting. The system comprises one or more entities configured to identify information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants. The at least one background object is associated with a first participant of the set of participants. The one or more entities are further configured to select, based on the identified information, the at least one background object to be provided in or to be excluded from the video stream from the first participant and shared with at least a second participant of the set of participants.
Advantageously, these aspects of the inventive concept maybe used in, for instance, video conferencing, for saving scarce network resources and for improved privacy settings for users of the video conferencing services.
Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.
Further features and advantages of the present teachings will become clear upon reading the following description and the accompanying drawings.
Brief description of the drawings
Figure l illustrates schematically an environment in which embodiments of the present teachings may be implemented.
Figure 2 is a flow chart of embodiments of a method in one or more entities.
Figure 3 is a flow chart of embodiments of a method in a managing unit.
Fig. 4 is a schematic diagram showing functional units of one or more entities according to an embodiment. Fig. 5 is a schematic diagram showing functional modules of one or more entities according to an embodiment.
Fig. 6 is a schematic diagram showing functional units of a managing unit according to an embodiment. Fig. 7 is a schematic diagram showing functional modules of a managing unit according to an embodiment.
Fig. 8 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.
Detailed description In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, interfaces, techniques, etc. in order to provide a thorough understanding. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description with unnecessary detail. Same reference numerals refer to same or similar elements throughout the description.
Web conferencing is an umbrella term for various types of online conferencing and collaborative services such as e.g. web seminars (webinars), webcasts and web meetings. This type of services is known under a variety of different names, such as online meeting, digital meeting, video meeting, e-meeting and virtual meeting. The present teachings are applicable to any such type of service.
What specific personal information a first user is comfortable with sharing may, for instance, depend on meeting context, e.g. whether it is private or business, on cultural aspects, or what may or may not be associated with trigger words in respective context. The relation to other meeting attendees may also be a factor to consider. For instance, in private life among trusted friends of known faiths, religion, etc., the first user may be comfortable sharing backdrop ornaments and would even like to put more details for certain objects (e.g. accentuate, emphasize) that are of high personal value, and such action may furthermore even serve to establish common ground among meeting participants, whereas on the other hand, in e.g. business context or in situations with unknow meeting participants, such personal details should be suppressed on perhaps even concealed entirely. In a mixed setup where a first user would be comfortable sharing some selected object visualization with a first meeting participant A, but for some reasons not with a second participant B, the currently commonly available solution is to either not send video to B at all or to conceal all details for both A and B.
The present disclosure provides methods and means for accomplishing this in a resource efficient way, i.e. to enable a user to select what to be shown to whom.
The present teachings provide means for a user and for an owner of a managing system to adaptively reduce the amount of consumed resources that are associated with transferring information, or “pixels”, corresponding to objects in the user’s digital meeting background.
Figure l illustrates schematically an environment in which embodiments of the present teachings may be implemented. A system l is shown, which comprises a cloud execution environment 2 wherein a number of network functions (NF) and dedicated functions (DF) reside. Such network functions may handle a variety of functions, e.g. mobility in a wireless and/ or wired network, security, routing. The network functions may be running on virtualized hardware and not be tied to specific hardware node, whereby they can be executed in different places depending on a scenario at hand and corresponding requirements. In the present disclosure one or more entities 7a, 7b may be implemented as such network functions, configured to perform embodiments of methods to be described, e.g. in relation to figure 2.
Further, a network infrastructure 3 is shown in figure 1. The network infrastructure 3 may be a network for handling wireless communication and various functions thereof. The network infrastructure 3 may then comprise a wired part connected to entities 7a, 7b of the cloud execution environment 2, and a wireless part comprising a wireless communications system 4, in turn comprising e.g. radio nodes 8a, 8b serving a number of wireless user equipment 5a, 5b. The radio nodes 8a, 8b may comprise user equipment 5a, 5b, relay nodes and access nodes. The access nodes are shown separated into antennas, radio units (RU) and digital units (DU). The network infrastructure 3 may support at least one 3GPP cellular communication standard, e.g. NB-IoT and the general 4th generation (4G) and 5th generation (5G) 3GPP wireless systems as well as any future related wireless networks from 3GPP wherein the skilled person would understand that the invention would be applicable, e.g. in the future 6G 3GPP network.
In an aspect, the one or more entities 7a, 7b may, in different embodiments, be located either in the cloud execution environment 2 or in the network infrastructure 3, e.g. in a managing server or a controlling node. The entities 7a, 7b, when comprising at least two, maybe distributed such that one or more entities is located in the cloud execution environment 2, and the remaining one or more in the network infrastructure 3.
The present disclosure suggests, in various embodiments, the use of machine- learning (ML) based identification and classification of backdrop/background objects associated with participants of a digital meeting and provides means for a managing unit 6 to assign more/fewer rendering resources to a selected inbound/outbound video stream in accordance with first-second-third users relations, and preferences of the meeting context. In context of a digital meeting hosting multiple users, a users’ device 5a, 5b may run a meeting application that is connected to at least one managing unit 6. The managing unit 6 may control inbound/outbound video streams to/from its managed users. The managing unit 6 maybe part of a managing server 9.
The managing server 9 may hold capabilities that dispatches radio interface resources among the cellular connected users in terms of e.g. QoS contracts, SLAs, resource allocation (i.e. scheduling of e.g. PRB allocation, etc.); a radio resource managing entity may typically be located in an eNB/gNB and maybe denoted as scheduler. The managing unit 6 may also hold capabilities of object identification and classification in terms of identifying/classifying objects in respective users’ background imagery conveyed from digital meeting applications running in user devices 5a, 5b.
The application and/or managing unit 6 may furthermore have capabilities of, for a specific identified and classified object, apart from determining its type, e.g. painting, furniture etc., also determine its money value, its corresponding trigger-value. Such trigger value may for instance comprise grade of controversial, “profanity”, etc. The application and/ or managing unit 6 may furthermore have a capability of substituting the specific object with a generic object of same type. Object values, etc., maybe obtained from external sources, e.g. internet servers, etc., located e.g. in the cloud environment 2.
The managing unit 6 may also hold capabilities of determining meeting participants inter-relations and context of the meeting. The managing unit 6 may, for example, identify which of the participants know each other prior to the meeting. The managing unit 6 may perform such identification e.g. through assessment of emails, chats, social medias, friend/VIP-settings, etc. Still further, the managing unit 6 may determine if the meeting is in a private or in a business/corporate context. Such determination may, e.g., be performed via email-addresses, based on time of day, and/ or meeting subject, etc. In a similar aspect, the managing entity 6 may by similar means assess individual participants emails, chats, social medias flows, friend interactions, web browsing patterns, etc. and classify users’ respective personal interest, preferences, etc.
The above determinations made by the managing unit 6 may thus be accomplished by the managing unit 6 comprising means for implementing a Machine Learning (ML)- based identification and classification of backdrop/background objects associated with.
Then managing unit 6 may deduct, for instance, one or more of:
- what objects in a first user’s background that may be of specific interest (or specific noninterest) for what set of other meeting participants;
- what objects in other meeting participant backgrounds that are of interest for the first user;
- and all cross-intersections between first, second, third users, etc.
In context of having determined objects-to-spectator relations and first user’s and spectators’ respective preferences, then the managing entity 6 may determine: what object that maybe given additional resolution, e.g. in combination with improved frame rate, color depth, etc., where the numbers of objects to accentuate typically is smaller than number of all possible objects in the user’s background image; - which other “complementary objects” in respective background that may be given less accentuating, in terms of e.g. lower frame rate, resolution or color depth, etc.;
- to which individual participant the video stream carrying said accentuated object should go (and implicitly, where said accentuated object should not go);
- a video encoding setting that corresponds to requirements to accentuate identified objects and corresponds to necessity to provide a lowest quality (e.g. resolution, etc.) to non-accentuated complementary background objects; in context of video (device) applications, full P-frames to update the image may not be used, but instead a smaller part that is selected dynamically depending on how much of the image that is being subject to changes between different frames. Typically, an arbitrary slice size and shape may be used; smallest entity for a macro block in H264 is typically in order or 8x8 (or 16x16) pixels.
Given that fewer areas in individual users’ background are identified as subject for accentuation, compared to all objects, and furthermore per-object identified to which specific meeting participant the accentuated object is intended, the less total radio resources are required for the managing server 9 to allocate to run the digital meeting. Thereby radio resources are made free, i.e. not allocated for the digital meeting purpose, and may therefore be used by the managing server 9 for other communication purposes.
Next a first scenario is described, for exemplifying the various aspects of the disclosed embodiments.
Three users participate in a digital meeting using a respective camera. User A likes watches and has several items on display in the back of the room, and also has a large window facing his garden.
User B likes gardening but does not care about watches.
User C likes watches but does not care for gardening.
Then in the meeting application of the managing unit 6, the managing unit 6 may hold information about the users A, B, C and in particular about their respective preferences. The managing unit 6 may detect and identify respective objects, in this example trees in the garden and the displayed watches in User A’s backdrop. The managing unit 6 may determine that video rendering effort may be spent on providing User B with better resolution on one set of selected preferred objects in A’s backdrop, e.g. trees and flowers, while User C is provided with better resolution on another set of preferred objects in A’s backdrop, e.g. watches. The managing unit 6 may provide respective receiving users with a video stream from a first user that has improved rendering/resolution on objects determined “liked”, and potentially the opposite on objects found “disliked”.
The first user does not have to figure out what in his/her backdrop that may be shown, may be non-disclosed and applied “more rendering to”, the managing entity 6 may more effort/less effort on different objects depending on individual receivers’ preference and sender/individual receivers’ relation.
A part of the managing unit 6 (e.g. a scheduling entity) may then determine new video encoding reflecting certain accentuated/non-accentuated objects, determine corresponding to the video encoding and QoS/SLA, determine a scheduling policy and associated radio interface resources, where said dedicated per user -user personalized media flow are provided to/from said individual users.
From the above example, it is clear that objects identified in the media stream maybe subject to various accentuation rules:
Objects subject to accentuation rules may, for instance, comprise book titles, paintings, furniture, etc.; human faces, e.g. family members; items, objects, etc. of interest determined from an ML model of recipient interests, and/or relevant to ongoing discussion of the ongoing meeting (or preceding meeting).
Accentuation rules may be applied manually according to first user’s preferences (“I’d like to emphasize my painting”), or they may be based on ML model identifying <triggerword> topics given profiles of meeting participants’ relation to “baseline”.
Given selected accentuation rule, the managing entity 6 may: apply the selected digital accentuation to a certain selected digital object. Such digital accentuation may comprise the managing server to render identified object with higher image/video quality, higher resolution/rate, etc. - improve objects visibility, readability and/ or objects relative relation to other non-accentuated objects in the media stream,
- provide user with notification of recipients' potential relation to identified object attributes: “Given that recipient A is known to like watches ... and that he is trusted ... I suggest that watch-objects to be rendered in 4k@9ofps”,
- Suggest user to apply action to adjust,
- Automatically adjust IF EstimatedComplianceValue > threshold
The managing entity 6 may provide objects relevant to media streams receivers' interest and preferences and/or objects relevant to ongoing discussion between communication parties
Next a second scenario is described, for exemplifying the various aspects of the disclosed embodiments.
Three users participate in a video meeting using respective cameras.
User A likes watches and paintings and have several on display in the room. User B is a trusted friend of User A.
User C is not a person trusted by User A but is someone that User A and User B is discussing a deal with about a used car.
Their respective devices, at least for User A, maybe used for scanning their respective rooms. For instance, perform a 3D scanning of places and objects. This can be performed using e.g. Lidar capabilities (e.g. determining ranges by targeting objects with a laser and measuring return time for reflected light), and creating an USDZ-file or similar, with a dictionary with location/ description of all identified backdrop objects. Further the device maybe used to create a reference room with the same type of objects. Then, for each object create an object-attribute-metric lookup, e.g. describing:
“value”, either from a public data base (e.g. watches: chrono24.com), or user- provided in an initial setup procedure, or user-prompted given that no value- object lookup is identified,
- “religious or political”, painting, book titles, symbols, posters, etc., “personal”, family portraits, heirloom, etc.
Next one or more attribute-to-disclosability tables are created, for instance according to:
Trust -> disclosable objects: all No trust -> disclosable objects: subset X
Political -> disclosable objects: subset Z Then in the meeting application, i.e. in the managing unit 6:
- it is determined that Users A and B have a trusted relation;
- it is determined that User A has no trust in User C; - respective objects (i.e. paintings and watches) in User A’s backdrop are detected and identified
The managing unit 6 may further determine that certain detected objects may be fully disclosed in User B’s media stream, that certain detected objects should not be disclosed in User C’s media stream. The managing unit 6 may further select disclosure rules according to User A’s no trust in User C, e.g. rules such as:
- Apply the selected digital overlay to a certain selected digital object,
- Where selected digital overlay to the selected digital object masks identity/attributes of said selected objects The managing unit 6 may further select disclosure rule according to A’s full trust in B where no digital object overlay at all is applied.
A part of the managing server (e.g. scheduling entity) may then determine new video encoding reflecting certain accentuated/non-accentuated objects, determine corresponding to said video encoding and QoS/SLA, determine a scheduling policy and associated radio interface resources, where said dedicated per user-user personalized media flow are provided to/from said individual users. Further details in context of partial object disclosure, full object alike-overlay are described in the following.
Objects identified in the media stream maybe subject to disclosure rules:
- where a conservative rule targets to show less detailed/ specific information on user’s background, substituting a first digital object associated with a first physical object with a second digital object having certain other attributes; e.g. exchanging the digital media representing a Monet painting with a digital overlay of a less valuable painting.
- where a less conservative rule targets to disclose more detailed/ specific information on user’s background
Objects subject to disclosure rules may comprise: Book titles, paintings, furniture, etc., human faces, e.g. family members.
The disclosure rules may be applied manually according to the first user’s preferences, and/or applied in accordance with relations between first user and other meeting participants (family, corporate, unknown), and/or based on an ML model identifying <triggerword> topics given profiles of meeting participants’ relation to “baseline”, and/or an ML model maybe trained with respect to potential meeting participants according to: cultures, religion, associated use & manners, gender, age, etc. Given selected disclosure rule, the managing entity 6 may:
Apply the selected digital overlay to a certain selected digital object
Provide user with notification of recipients' potential relation to identified object attributes
Suggest user to apply action to adjust, or Automatically adjust IF EstimatedComplianceValue > threshold
Another aspect of the disclosure is to have a local application in the user device that analyzes background objects and/or sounds. The local application may use known preferences related to each participant in an online meeting to prioritize and/ or remove part of background information that may not be relevant for all participants. In other embodiments, the local application may use known preferences related to each participant in an online meeting to prioritize and share same background information to all participants; this step may require that individual local applications are communicating, and share said user preferences. In still other embodiments the application may prioritize background information and share part that are relevant for each participant.
Various embodiments have been described hitherto that, with ML-based identification and classification of background objects associated with participants of digital meeting, provides means for a managing entity 6 to assign more or fewer rendering resources to a selected inbound/outbound video stream in accordance with first-second-third users relations, preferences or meeting context. Based on this mechanism for identification and selection of focus-objects in user-user media streams for a digital meeting, video encoding resources and implicitly air interface resources to convey said media stream, may be spent only in user-user connection in which at least one user benefit from having certain objects disclosed, so that another part of managing node (scheduling entity) may save resources by efficiently allocate resources to video streams that convey relevant objects and that said objects are determined relevant on per-user individual basis. Hence, that different high- resolution objects are being conveyed only between individual communication parties according to determined user preferences, and that scheduling entity in total may assign less resources to said digital meeting.
Figure 2 is a flow chart of embodiments of a method 20 for selecting at least one background object for a video stream between at least two participants of a digital meeting. The method 20 is performed in one or more entities of a system 1 as described in relation to figure 1.
A method 20 is provided for selecting one or more background objects for a video stream between two or more participants of a digital meeting. The selection method 20 is performed by one or more entities 7a, 7b. The method 20 comprises identifying 22 information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants. The background object is associated with a first participant of the set of participants. The method 20 further comprises selecting 24, based on the identified information, the at least one background object to be provided or to be excluded in the video stream from the first participant and to be shared with at least a second participant of the set of participants. In an embodiment, the identifying 22 comprises using machine learning for identifying and classifying 23 background objects associated with at least one of the two or more participants.
In different embodiments, the identifying 22 comprises identifying the at least one background object based on one or more of: type of object, economic value of object and grade of controversiality of object.
In various embodiments, the identifying 22 comprises identifying the at least one background object based on one or more of: preferences of the first participant, preferences of the at least second participant, central analysis of controversiality of the object and central analysis of economic value of the object. In various embodiments, the selecting 24 comprises selecting the at least one background object based on one or more of: relation of the first participant to the at least second participant, meeting context, degree of interest for the one or more background objects and cross-intersection between three or more of the set of participants. In various embodiments, the method 20 comprises providing 26 the information on the one or more background objects to a managing unit 6 for managing use of resources of the conveying of the video stream.
In various embodiments, the identifying 22 and selecting 24 are performed in a single entity of a cloud execution environment 2. Figure 3 is a flow chart of embodiments of a method in managing unit 6. The method 30 is performed in a managing unit 6 for managing communication resources of a video stream between two or more participants of a digital meeting. The method 30 comprises obtaining 32 information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the at least one background object being associated with a first participant of the set of participants. The method 30 further comprises assigning 34 resources to the first participant based on the obtained information.
In an embodiment, the obtaining 32 comprises receiving the information on the one or more background objects from one or more entities 7a, 7b of a cloud execution environment 2.
In various embodiments, the obtained information comprises determined and classified background object preferences for at least the first and the second participants of the set of participants. The method 30 then comprises determining, based on the background object preferences, one or more of: resolution, frame rate and color depth, based on degree of accentuation determined for a respective participant of the set of participants.
In various embodiments, the assigning 34 comprises assigning to the first participant amount of radio interface resources for one or more video streams based on one or more of: determined accentuating or non-accentuating video encoding, and determined per-user radio resource demand in accordance to video-encoding.
In various embodiments, the resources comprise radio resources required for at least part of a path of the video stream.
In various embodiments, the obtained information comprises one or more of: relation of the first participant to the at least second participant, meeting context, degree of interest of the at least second participant for the at least one background object and cross-intersection between three or more of the set of participants.
In various embodiments, the managing unit 6 comprises a node of a network infrastructure 3 or a node 8a, 8b of an access network of a wireless communications network. Fig. 4 schematically illustrates, in terms of a number of functional units, the components of one entity of the one or more entities according to an embodiment. Processing circuitry 110 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 500a (as in Fig. 8), e.g. in the form of a storage medium 520. The processing circuitry no may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).
Particularly, the processing circuitry no is configured to cause the entity to perform a set of operations, or steps, as disclosed above. For example, the storage medium 130 may store the set of operations, and the processing circuitry no may be configured to retrieve the set of operations from the storage medium 130 to cause the entity to perform the set of operations. The set of operations maybe provided as a set of executable instructions. The processing circuitry no is thereby arranged to execute methods as herein disclosed. The storage medium 230 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.
The entity may further comprise a communications interface 120 at least configured for communications with other entities, functions, nodes, and devices. As such the communications interface 120 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 110 controls the general operation of the entity e.g. by sending data and control signals to the communications interface 120 and the storage medium 130, by receiving data and reports from the communications interface 120, and by retrieving data and instructions from the storage medium 130. Other components, as well as the related functionality, of the entity are omitted in order not to obscure the concepts presented herein.
Fig. 5 is a schematic diagram showing functional modules of one or more entities according to an embodiment. Fig. 5 schematically illustrates, in terms of a number of functional modules, the components of an entity according to an embodiment. The entity of Fig. 5 comprises a number of functional modules; an identify module 210 configured to perform step 22, and a select module 220 configured to perform step 24. The entity of Fig. 5 may further comprise a number of optional functional modules, such as a provide module 230 configured to perform step 26. In general terms, each functional module 210, 220, 230 may be implemented in hardware or in software. Preferably, one or more or all functional modules 210, 220, 230 maybe implemented by the processing circuitry no, possibly in cooperation with the communications interface 120 and the storage medium 130. The processing circuitry 110 may thus be arranged to from the storage medium 130 fetch instructions as provided by a functional module 210, 220, 230 and to execute these instructions, thereby performing any steps of the entity as disclosed herein. Fig. 6 is a schematic diagram showing functional units of a managing unit 6 according to an embodiment. Processing circuitry 310 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 500b (as in Fig. 8), e.g. in the form of a storage medium 520. The processing circuitry 510 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).
Particularly, the processing circuitry 310 is configured to cause the managing unit 6 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 330 may store the set of operations, and the processing circuitry 310 may be configured to retrieve the set of operations from the storage medium 330 to cause the managing unit 6 to perform the set of operations. The set of operations maybe provided as a set of executable instructions. The processing circuitry 310 is thereby arranged to execute methods as herein disclosed. The storage medium 330 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.
The managing unit 6 may further comprise a communications interface 320 at least configured for communications with other entities, functions, nodes, and devices. As such the communications interface 320 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 310 controls the general operation of the entity e.g. by sending data and control signals to the communications interface 320 and the storage medium 330, by receiving data and reports from the communications interface 320, and by retrieving data and instructions from the storage medium 330. Other components, as well as the related functionality, of the entity are omitted in order not to obscure the concepts presented herein. Fig. 7 schematically illustrates, in terms of a number of functional modules, the components of a managing unit according to an embodiment. The managing unit of Fig. 7 comprises a number of functional modules; an provide module 410 configured to perform step 32, and an assign module 420 configured to perform step 34. The managing unit of Fig. 5 may further comprise a number of optional functional modules to perform additional steps. In general terms, each functional module 410, 420 maybe implemented in hardware or in software. Preferably, one or more or all functional modules 410, 420 may be implemented by the processing circuitry 310, possibly in cooperation with the communications interface 320 and the storage medium 330. The processing circuitry 310 may thus be arranged to from the storage medium 330 fetch instructions as provided by a functional module 410, 420 and to execute these instructions, thereby performing any steps of the managing unit as disclosed herein.
Fig. 8 shows one example of a computer program product 500a, 500b comprising computer readable means 520. On this computer readable means 520, a computer program 510a can be stored, which computer program 510a can cause the processing circuitry 110 and thereto operatively coupled entities and devices, such as the communications interface 120 and the storage medium 130, to execute methods according to embodiments described herein. The computer program 510a and/or computer program product 500a may thus provide means for performing any steps of the one or more entities as herein disclosed. On this computer readable means 520, a computer program 510b can be stored, which computer program 510b can cause the processing circuitry 310 and thereto operatively coupled entities and devices, such as the communications interface 320 and the storage medium 330, to execute methods according to embodiments described herein. The computer program 510b and/ or computer program product 500b may thus provide means for performing any steps of the managing unit as herein disclosed.
In the example of Fig. 8, the computer program product 500a, 500b is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu- Ray disc. The computer program product 500a, 500b could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 510a, 510b is here schematically shown as a track on the depicted optical disk, the computer program 510a, 510b can be stored in any way which is suitable for the computer program product 500a, 500b.
The invention has mainly been described herein with reference to a few embodiments. However, as is appreciated by a person skilled in the art, other embodiments than the particular ones disclosed herein are equally possible within the scope of the invention, as defined by the appended patent claims.

Claims

Claims
1. A method (20) for selecting one or more background objects for a video stream between two or more participants of a digital meeting, the selection method (20) being performed by one or more entities (7a, 7b), and comprising: - identifying (22) information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the background object being associated with a first participant of the set of participants, and - selecting (24), based on the identified information, the at least one background object to be provided or to be excluded in the video stream from the first participant and to be shared with at least a second participant of the set of participants.
2. The method (20) as claimed in claim 1, wherein the identifying (22) comprises using machine learning for identifying and classifying (23) background objects associated with at least one of the two or more participants.
3. The method (20) as claimed in claim 1 or 2, wherein the identifying (22) comprises identifying the at least one background object based on one or more of: type of object, economic value of object and grade of controversiality of object.
4. The method (20) as claimed in any of the preceding claims, wherein the identifying (22) comprises identifying the at least one background object based on one or more of: preferences of the first participant, preferences of the at least second participant, central analysis of controversiality of the object and central analysis of economic value of the object.
5. The method (20) as claimed in any of the preceding claims, wherein the selecting (24) comprises selecting the at least one background object based on one or more of: relation of the first participant to the at least second participant, meeting context, degree of interest for the one or more background objects and cross intersection between three or more of the set of participants.
6. The method (20) as claimed in any of the preceding claims, comprising: - providing (26) the information on the one or more background objects to a managing unit (3) for managing use of resources of the conveying of the video stream.
7. The method (20) as claimed in any of the preceding claims, wherein the identifying (22) and selecting (24) are performed in a single entity of a cloud execution environment (2).
8. The method (20) as claimed in any of the preceding claims, wherein the identifying (22) is performed in a first entity of a cloud execution environment and the selecting is performed in a second entity of a cloud execution environment (2).
9. Computer program (510a) comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method (20) according to any of claims 1-8.
10. A computer program product (520a) comprising a computer program (510a) as claimed in claim 9 and a computer readable means on which the computer program (510a) is stored.
11. A method (30) in a managing unit (6) for managing communication resources of a video stream between two or more participants of a digital meeting, the method
(30) comprising:
- obtaining (32) information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the at least one background object being associated with a first participant of the set of participants, and
- assigning (34) resources to the first participant based on the obtained information.
12. The method (30) as claimed in claim 11, wherein the obtaining (32) comprises receiving the information on the one or more background objects from one or more entities of a cloud execution environment (2).
13. The method (30) as claimed in claim 11 or 12, wherein the obtained information comprises determined and classified background object preferences for at least the first and the second participants of the set of participants, and the method (30) comprising determining, based on the background object preferences, one or more of: resolution, frame rate and color depth, based on degree of accentuation determined for a respective participant of the set of participants.
14. The method (30) as claimed in any of claims 11 - 13, wherein the assigning (34) comprising assigning to the first participant amount of radio interface resources for one or more video streams based on one or more of: determined accentuating or non-accentuating video encoding, and determined per-user radio resource demand in accordance to video-encoding.
15. The method as claimed in any of claims 11 - 14, wherein the resources comprise radio resources required for at least part of a path of the video stream. 16. The method (30) as claimed in any of claims 11 - 15, wherein the obtained information comprises one or more of: relation of the first participant to the at least second participant, meeting context, degree of interest of the at least second participant for the at least one background object and cross-intersection between three or more of the set of participants. 17. The method (30) as claimed in any of claims 11 - 16, wherein the managing unit
(6) comprises a node of a network infrastructure (3) or a node of an access network (8a, 8b) of a wireless communications network (1).
18. Computer program (510b) comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according to any of claims 11- 17.
19. A computer program product (500b) comprising a computer program (510b) as claimed in claim 18 and a computer readable means on which the computer program (510b) is stored.
20. A system (1) configured to select background objects for a video stream between two or more participants of a digital meeting, comprising one or more entities (7a,
7b) configured to:
- identify information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the at least one background object being associated with a first participant of the set of participants, and
- select, based on the identified information, the at least one background object to be provided in or to be excluded from the video stream from the first participant and shared with at least a second participant of the set of participants. 21. The system (1) as claimed in claim 20, comprising a plurality of entities (7a, 7b) and wherein a first entity (7a) is configured to identify information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the at least one background object being associated with a first participant of the set of participants, and to select, based on the identified information, the at least one background object to be provided in the video stream from the first participant and shared with at least a second participant of the set of participants.
22. The system (l) as claimed in claim 20, comprising a plurality of entities and wherein a first entity (7a) is configured to identify the information on at least one background object of a background captured by a camera and to be shared between two or more participants of a set of participants, the at least one background object being associated with a first participant of the set of participants, and a second entity (7b) is configured to select based on the identified information, the at least one background object to be provided in the video stream from the first participant and shared with at least a second participant of the set of participants.
PCT/EP2021/058002 2021-03-26 2021-03-26 Method for selecting one or more background objects for a video stream WO2022199853A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/EP2021/058002 WO2022199853A1 (en) 2021-03-26 2021-03-26 Method for selecting one or more background objects for a video stream

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2021/058002 WO2022199853A1 (en) 2021-03-26 2021-03-26 Method for selecting one or more background objects for a video stream

Publications (1)

Publication Number Publication Date
WO2022199853A1 true WO2022199853A1 (en) 2022-09-29

Family

ID=75362601

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2021/058002 WO2022199853A1 (en) 2021-03-26 2021-03-26 Method for selecting one or more background objects for a video stream

Country Status (1)

Country Link
WO (1) WO2022199853A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020196848A1 (en) * 2001-05-10 2002-12-26 Roman Kendyl A. Separate plane compression
US20180176508A1 (en) * 2016-12-20 2018-06-21 Facebook, Inc. Optimizing video conferencing using contextual information
US10554931B1 (en) * 2018-10-01 2020-02-04 At&T Intellectual Property I, L.P. Method and apparatus for contextual inclusion of objects in a conference
US10580149B1 (en) * 2017-06-26 2020-03-03 Amazon Technologies, Inc. Camera-level image processing
US20200211201A1 (en) * 2018-12-28 2020-07-02 Cyberlink Corp. Systems and methods for foreground and background processing of content in a live video

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020196848A1 (en) * 2001-05-10 2002-12-26 Roman Kendyl A. Separate plane compression
US20180176508A1 (en) * 2016-12-20 2018-06-21 Facebook, Inc. Optimizing video conferencing using contextual information
US10580149B1 (en) * 2017-06-26 2020-03-03 Amazon Technologies, Inc. Camera-level image processing
US10554931B1 (en) * 2018-10-01 2020-02-04 At&T Intellectual Property I, L.P. Method and apparatus for contextual inclusion of objects in a conference
US20200211201A1 (en) * 2018-12-28 2020-07-02 Cyberlink Corp. Systems and methods for foreground and background processing of content in a live video

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
AMIN MOHAMMED ARIF ET AL: "A review of mobile cloud computing architecture and challenges to enterprise users", 2013 7TH IEEE GCC CONFERENCE AND EXHIBITION (GCC), IEEE, 17 November 2013 (2013-11-17), pages 240 - 244, XP032544587, ISBN: 978-1-4799-0722-9, [retrieved on 20140108], DOI: 10.1109/IEEEGCC.2013.6705783 *

Similar Documents

Publication Publication Date Title
US10630791B2 (en) Customized presentation of event guest lists in a social networking system
CN105376442B (en) Cloud system and the method that content is shown by cloud system
US10757366B1 (en) Videoconferencing dynamic host controller
US20170171267A1 (en) Systems and methods to present responses to an event in a social network
US8826150B1 (en) System and method for tagging images in a social network
CN108702481B (en) System and method for creating composite video
US10510165B2 (en) Systems and methods for context based image compression
US9942516B1 (en) Optimizing video conferencing using contextual information
US20140229541A1 (en) Method and system for content management of social networking data
US20220286418A1 (en) Dynamically controlled permissions for managing the communication of messages directed to a presenter
US10579898B2 (en) Systems and methods for provisioning content using barrel projection representation
Wong et al. 360-degree video bandwidth reduction: Technique and approaches comprehensive review
US20190207993A1 (en) Systems and methods for broadcasting live content
US20170052650A1 (en) Event media management systems and methods
US20200007597A1 (en) Systems and methods for optimizing simulcast streams in group video calls
WO2022199853A1 (en) Method for selecting one or more background objects for a video stream
US11277275B2 (en) Device ranking for secure collaboration
US20200193389A1 (en) Providing insights about attendees of smart web-events
US20220092332A1 (en) Systems and methods for provisioning content
US20140344350A1 (en) Image Session Invitation and Management Techniques
US20230291776A1 (en) Group visualizations in online meetings
US20240064183A1 (en) Flexible orchestration of session content in venue networks
Choi et al. Towards zero-configuration in device collaboration using device sociality
Dolog et al. Decentralized, agent based and social approaches to user modelling
WO2016056042A1 (en) Automatic group generation method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21716127

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21716127

Country of ref document: EP

Kind code of ref document: A1